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In  1991  the  International  Military  Education  and  Training  (IMET)  program 

was  expanded  to  include  training  programs  focusing  on  civilian  control  over  the 

military,  respect  for  human  rights,  and  responsible  defense  resource  management. 

In    1993    Congress   institutionalized  the   federal   government's   commitment   to 

performance  measurement  by  signing  into  law  the  Government  Performance  and 

Results  Act  (GPRA).    GPRA  requires  the  Departments  of  State  and  Defense  to 

demonstrate  the  results  achieved  by  programs  such  as  Expanded  IMET  (E-IMET). 

The  purpose  of  this  study  is  to  address  how  the  Defense  Security  Cooperation 

Agency  (DSCA)  can  tackle  the  challenge  of  measuring  the  effectiveness  of  the  E- 

IMET  program.  A  spectrum  of  approaches  exists  to  evaluate  public  programs  and 

is  anchored  on  one  end  by  the  "technically  rational  paradigm"  and  on  the  opposite 

end  by  the  "politically  rational  paradigm."    By  organizing  the  security  assistance 

objectives  of  key  E-IMET  stakeholders  into  a  Global  Hierarchy  the  researcher  was 

able  to  link  the  E-IMET  program  to  national  level  goals,  creating  an  objective 

baseline  from  which  to  measure  the  performance  of  the  program.     Individual 

objectives    hierarchies    were    then    created    for    each    E-IMET    objective    and 

performance   indices  were  proposed  to  meet  the  requirements   of  the   GPRA 

mandate. 
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I.  INTRODUCTION 

A.    BACKGROUND 

1.  Security  Assistance 

Security  assistance,  as  defined  in  its  simplest  terms,  concerns  the  transfer  of 
military  and  economic  assistance  products  and  services  to  foreign  governments  through 
sale,  grant,  lease  or  loan.  Promulgated  into  law  by  the  Foreign  Assistance  Act  of  1961. 
security  assistance  includes  the  Foreign  Military  Sales  (FMS)  program,  the  Economic 
Support  Fund  (ESF),  International  Military  Education  and  Training  (IMET), 
Peacekeeping  operations,  and  the  Commercial  Export  Sales  program.  Currently  the 
Secretary  of  State  has  statutory  responsibility  for  supervision  and  general  direction  of 
security  assistance  programs  and  the  Secretary  of  Defense  is  responsible  for 
implementing  military  programs  through  the  Defense  Security  Cooperation  Agency 
(DSC  A).1 

2.  Post  Cold  War  Security  Environment 

Recent  world  events  such  as  the  break-up  of  the  former  Soviet  Union  and  Gulf 
War  have  caused  a  modification  of  U.S.  security  assistance  goals.  According  to  a  1997 
U.S.  General  Accounting  Office  (GAO)  report. 


1  Until  October  1,  1998.  DSCA  was  known  as  the  Defense  Security  Assistance  Agency  (DSAA). 

1 


We  now  face  challenges  involved  in  NATO  expansion,  assisting  the 
transition  of  former  Communist  countries,  integrating  China  into  the  world 
economic  system,  reforming  the  United  Nations  and  the  World  Bank, 
building  a  credible  World  Trade  Organization,  and  attacking  the  menace 
of  illegal  drugs  and  AIDS.  These  events  necessitate  a  careful  rethinking  of 
the  programs,  approaches,  and  activities  at  the  U.S.  government's  disposal 
to  advance  its  interests.  At  the  same  time,  the  current  environment  also 
affords  the  chance  to  reassess  programs  and  activities  on  their  merits  and 
determine  if  they  are  relevant  in  today's  world  (GAO,  1997,  p.  1). 

Security  assistance  programs  such  as  IMET  focus  upon  the  enhancement  of 

democratic  institutions  worldwide.    With  the  new  emphasis  on  providing  support  for 

emerging   democracies,   a   challenge   facing   policy   makers    in   the   Post   Cold    War 

environment  is  to  assess  the  effectiveness  of  current  security  assistance  programs. 

3.  IMET  and  E-IMET 

a.  The  IMET  Program 

Section  #541  of  the  Foreign  Assistance  Act  (FAA)  of  1961  defines  IMET 
as  a  key  component  of  U.S.  security  assistance  that  provides  professional  military 
education  (PME)  on  a  grant  basis  to  students  from  allied  and  friendly  nations  (GAO, 
1992).  The  Congressional  Presentation  for  Foreign  Operations  (CPFO)  states  that  the 
overall  goal  of  the  program  is  to  "improve  the  ability  of  participating  foreign  countries  to 
utilize  their  resources"  thereby  contributing  to  greater  self-reliance  (CPFO,  1998,  p.  19). 
The  intention  is  to  provide  a  wide  range  of  benefits  to  the  U.S  in  terms  of  collective 
security,  stability,  and  peace. 

IMET  is  a  program  that,  for  a  relatively  modest  investment,  presents 
democratic  alternatives  to  key  foreign  military  and  civilian  leaders  (CPFO,  1998).  The 
IMET  program's  FY98  budget  request  was  for  a  mere  50  million  dollars,  less  than  the 


cost  of  one  U.S.  Air  Force  F-22  fighter  jet.2  In  fact,  funding  for  the  entire  international 
affairs  budget  (Federal  Budget  Function  #150),  which  totaled  $18.1  billion  in  fiscal  year 
1997.  constitutes  only  one  percent  of  the  federal  budget  and  just  three  to  four  percent  of 
federal  discretionary  funding  (GAO,  1997).  Funding  for  the  international  affairs  account, 
however,  has  declined  in  real  terms  about  6%  annually  from  1992  to  1997  (GAO.  1997). 
Nineteen  ninety-seven  funding  was  23%  lower  than  1992  levels  (GAO.  1997). 
Additionally,  Function  150  is  slated  to  receive  cuts  of  over  13%  in  real  terms  by  2002 
(GAO,  1997). 

Although  international  affairs  programs  such  as  IMET  comprise  a  fraction 
of  federal  discretionary  funding,  "they  are  designed  to  influence  world  political  and 
economic  agendas"  (GAO,  1997,  p.  2).3 

b.  The  Expanded  IMET  Initiative 

IMET  is  expanding  and  taking  new  directions  in  response  to  the  changing 
global  political  scene.  Significant  changes  have  taken  place  to  align  the  program 
objectives  with  U.S.  foreign  policy  interests  in  the  post-Cold  War  environment.  In  1991 
the  Congress  amended  the  1961  Foreign  Assistance  Act  with  the  Foreign  Operations 
Appropriations  Act,  earmarking  $1  million  of  IMET  funds  to  expand  the  IMET  program. 
This  initiative  expanded  existing  IMET  courses  and  implemented  new  training  programs 


2  Currently  the  Air  Force  estimates  that  the  F-22  fighter  jet  will  cost  roughly  $87  million  each  (Cole. 
1993). 

->  According  to  the  Congressional  Budget  Office  (CBO).  1997  discretionary  outlays  totaled  approximately 
$550  billion  (CBO,  1997).  Therefore,  funding  for  Budget  Function  =150  equaled  roughly  3%  of  all  1997 
discretionary  expenditures  (GAO,  1997).  The  1997  IMET  appropriation  of  $43.5  million  only  absorbed 
.24%  of  Function  #150  funding,  or  less  than  .01%  of  total  1997  discretionary  spending. 


focusing  on  civilian  control  over  the  military,  respect  for  human  rights,  and  responsible 
defense  resource  management  (U.S.  Congress,  1991).  Most  importantly,  however,  the 
traditional  IMET  student  population  was  "enlarged  to  include  civilian  personnel  from 
non-defense  ministries,  officials  from  the  legislative  branch  who  deal  with  military 
matters,  and  personnel  from  nongovernmental  agencies  having  defense-related  interests" 
(Cope,  1995,  p.  6).  By  1998  approximately  25-30%  of  total  IMET  funding  was  dedicated 
to  the  E-IMET  initiative  (Verry,  1998). 

c.  IMET  and  E-IMET:  A  Low-Cost  Alternative 

In  1990,  DSCA  claimed  that  IMET  is  an  effective,  low-cost  component  of 
the  global  U.S.  security  assistance  effort  (Larson,  1993).  "The  IMET  program  has 
become  one  of  the  most  cost-effective  means  for  promoting  U.S.  interests  and  advancing 
security  in  the  Asian-Pacific-Indian  Ocean  region  and  throughout  the  world,"  confirmed 
the  Commander-in-Chief  Pacific  in  1993  (Larson,  1993,  p.  14).  There  is  a  belief  among 
existing  security  assistance  providers  that  the  relatively  small  cost  of  programs  such  as 
IMET  and  E-IMET  provides  a  return  for  U.S.  objectives  on  a  global  scale  and  that  over 
the  years  this  return  is  far  greater  than  the  original  investment. 

4.  Results  Oriented  Emphasis  for  Security  Assistance  and  Government 

According  to  SECSTATE  Christopher  in  1993, 

The  watchwords  for  security  assistance  are  now  empowerment,  partner- 
ship, and  effectiveness.  Our  foreign  assistance  programs  will  be  result- 
oriented,  not  expenditure-oriented.  Where  scarce  development  resources 
cannot  be  used  effectively,  our  assistance  programs  should  be  reduced  or 
redirected.  Our  development  assistance  should  be  judged  not  on  the  basis 
of  funds  obligated  but  on  the  basis  of  results  achieved,  and  the  same 
applies  to  security  assistance  (Christopher,  1993). 


Richard  Moose,  Under  Secretary  for  Management  in  1994.  echoed  the  Secretary's 
words  and  cited  the  fact  that  the  Foreign  Assistance  Act  (FAA)  of  1961  has  become  laden 
with  competing  and  conflicting  goals  and  objectives. 

In  a  time  of  reduced  budgets,  it  is  more  important  than  ever  that  our 
programs  be  reformed  and  our  scarce  resources  be  targeted  accurately  and 
used  effectively.  We  cannot  afford  waste  and  inefficiency.  We  [must] 
shift  our  focus  away  from  how  we  do  things  and  toward  what  we  are 
trying  to  accomplish — away  from  process  toward  results.  We  will  set 
clearly  understood  goals  by  which  our  programs  can  be  measured,  and  we 
will  demand  measurable  results  (Moose,  1994). 

Two  particular  initiatives  that  are  national  in  scope  and  demonstrate  overall 
government  emphasis  on  performance  measurement  are  the  Government  Performance 
and  Results  Act  (GPRA)  of  1993  and  the  Service  Efforts  and  Accomplishments  (SEA) 
Reporting  initiative  of  the  Governmental  Accounting  Standards  Board  (GASB). 

Signed  by  President  Clinton  in  1993,  the  GPRA  institutionalized  the  federal 
government's  commitment  to  developing  and  deploying  effective  performance  measure- 
ment and  performance  management  systems  based  on  agency  goals  and  objectives.  The 
Results  Act  requires  all  government  agencies  (except  the  Central  Intelligence  Agency  and 
some  very  small  agencies)  to  establish  top-level  goals  and  objectives,  define  how  they 
intend  to  achieve  those  goals,  and  demonstrate  how  they  will  measure  agency  and 
program  performance  in  achieving  those  goals  (GPRA,  1993).  Agencies  are  required  to 
submit  annual  performance  reports  to  Congress  starting  no  later  than  March  31,  2000. 
comparing  actual  performance  against  agency  plans  (Epstein,  1996). 


While  GASB  actions  do  not  have  the  force  of  law  like  GPRA,  their  SEA 
Reporting  initiative  is  recognized  by  the  American  Institute  of  Certified  Public 
Accountants  for  use  in  public  financial  reporting  (Epstein,  1996).  With  SEA  reporting, 
GASB  has  moved  beyond  standard  monetary  accounting  into  the  realm  of  performance 
measurement.  The  SEA  framework  stresses  outcomes  and  efficiency  as  well  as  inputs 
and  outputs  (Epstein,  1996).  In  short,  GASB  has  recognized  "that  public  accountability 
cannot  be  achieved  only  through  traditional  reporting  of  monetary  information"  (Epstein, 
1996,  p.  62).  Performance  reporting  is  also  needed. 

B.         STATEMENT  OF  PROBLEM 

In  this  time  of  reduced  budgets  and  evolving  security  assistance  and  performance 
review  initiatives,  the  DSCA  and  the  Departments  of  State  (DoS)  and  Defense  (DoD)  are 
faced  with  the  challenge  of  measuring  the  effectiveness  and  performance  of  their  security 
assistance  programs.  How  this  is  to  be  done  remains  an  open  question. 

A  1 992  GAO  report  exploring  post-Cold  War  security  assistance  program  changes 
highlighted  the  fact  that  the  DoD  and  DoS  had  not  yet  agreed  on  a  format  or  the  type  of 
information  to  be  included  in  an  assessment  of  the  overall  IMET  program  (GAO,  1992). 
DSCA  initially  noted  that  the  success  of  the  E-IMET  initiative  could  not  be  measured  for 
"at  least  another  5  years"  (GAO,  1992).  GAO  recommended  that  the  Departments  of 
State  and  Defense  jointly  develop  a  system  to  evaluate  the  effectiveness  of  the  E-IMET 
program. 


C.  RESEACH  QUESTIONS 

The  primary  research  question  of  this  thesis  is:  What  could  a  performance 
evaluation  system  developed  for  the  E-IMET  program  look  like? 

Subsidiary  research  questions  supporting  this  issue  are: 

1 .  What  are  the  challenges  to  performance  evaluation  of  public  programs? 

2.  What  factors  unique  to  the  political  environment  necessarily  limit  the 
application  of  rational-analytic  assessment  models  when  measuring  costs 
and  benefits  of  current  security  assistance  programs? 

3.  Can  a  system  be  developed  that  balances  the  reality  of  the  political 
bureaucratic  process  with  the  rationality  of  economic  cost-benefit 
analysis? 

4.  What  are  the  implications  of  implementing  a  performance  evaluation 
system  on  current  security  assistance  programs? 

D.  OBJECTIVE 

The  objective  of  this  thesis  is  to  address  how  the  DSC  A  and  the  Department  of 
Defense  can  tackle  the  challenge  of  measuring  the  effectiveness  and  performance  of  their 
security  assistance  programs.  Specifically,  it  will  investigate  potential  methods  for 
performance  evaluation  of  the  E-IMET  program. 

E.  METHODOLOGY 

The  following  approach  will  be  used  during  this  study: 

1 .  Identify'  a  continuum  of  policy  and  program  evaluation  paradigms  within 
the  public  sector  performance  evaluation  literature. 

2.  Identify  the  key  stakeholders  of  the  E-IMET  security  assistance  program 
and  the  objectives  and  goals  of  each. 

3.  Develop  a  Global  Objectives  Hierarchy  for  U.S.  security  assistance. 
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4.  Interview  members  of  the  IMET  and  E-IMET  management  staff  to 
validate  the  proposed  objectives  hierarchy. 

5.  Develop  performance  indicators  for  the  E-IMET  program. 

6.  Address  the  impact  of  the  proposed  evaluation  methodology  on  the  current 
E-IMET  program. 

7.  Recommend  a  course  of  action  that  brings  the  E-IMET  program  in  line 
with  the  federal  government's  mandate  for  performance  review. 

F.         ORGANIZATION  OF  STUDY 

Chapter  I,  Introduction,  will  present  the  focus  of  this  study.  It  will  introduce  the 
IMET  and  E-IMET  security  assistance  programs  and  discuss  the  challenges  presented  by 
the  changing  international  security  environment  and  the  1993  Government  Performance 
and  Results  Act. 

Chapter  II,  Literature  Review,  will  present  the  purpose,  current  utilization,  and 
future  use  of  performance  measurement  in  government.  It  will  also  explore  the 
challenges  of  public  sector  performance  evaluation  and  conclude  with  a  description  of  the 
process  that  will  be  used  to  measure  the  effectiveness  of  the  E-IMET  program. 

Chapter  III,  Methodology,  will  outline  the  research  methodology  used  to  conduct 
this  study. 

Chapter  IV,  Data  Collection,  will  identify  the  stakeholders  of  the  E-IMET 
program  in  order  to  inform  the  objectives  setting  process  and  enable  the  development  of 
performance  indicators. 


Chapter  V,  Analysis:  Part  I.  will  identify  each  stakeholder's  security  assistance 
objectives  in  order  to  form  a  Global  Objectives  Hierarchy  and  create  an  objective  baseline 
for  future  performance  measurement  efforts. 

Chapter  VI,  Analysis:  Parr  II.  will  propose  various  performance  indicators  for  the 
E-I MET  program. 

Chapter  VII,  Conclusion,  will  summarize  the  researcher's  findings  and 
recommend  a  course  of  action  for  the  implementation  of  an  E-IMET  performance 
evaluation  svstem. 
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II.         LITERATURE  REVIEW 

A.         PERFORMANCE  EVALUATION  IN  THE  PUBLIC  SECTOR 

Much  of  the  existing  literature  pertaining  to  performance  evaluation  in  the  public 
sector  explores  the  purpose,  current  utilization,  implementation  problems,  and  future  use 
of  performance  measurement  in  government.  In  order  to  fully  understand  performance 
evaluation  one  must  clearly  define  a  measurement  system,  differentiate  between  process 
and  results-oriented  systems,  recognize  the  purpose  and  need  for  a  system,  and  appreciate 
the  forces  affecting  its  application. 

1.  Measurement  System  Defined 

According  to  Halachmi  and  Bouckaert,  a  measurement  system  "consists  of 
practices,  procedures,  criteria,  and  standards  that  govern  the  collection  of  data  (input),  the 
analysis  of  the  data  (throughput),  and  the  compilation  of  the  results  into  quantitative  or 
qualitative  forms  (output)'"  (Halachmi  and  Bouckaert,  1996,  p.  2).  Greiner  cites  a  1992 
GAO  report  that  further  breaks  performance  measurement  into  subsystems: 

Program  performance  measurement  is  commonly  defined  as  the  regular 
collection  and  reporting  of  a  range  of  data  including: 

Inputs  (dollars,  staff,  and  materials). 

Workload  or  activity  levels, 

Outputs  or  final  products. 

Outcomes  of  products  or  services,  and 

Efficiency-cost  per  unit  output  or  output  per  unit  cost,  sometimes  referred 
to  as  "productivity"  (Greiner,  1996.  p.  2). 
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This  systems  definition,  including  inputs,  throughput,  outputs,  and  outcomes, 
must  also  recognize  factors  that  influence  each  subsystem.  For  instance,  the  input  side  is 
influenced  by  criteria  and  standards,  the  methods  and  frequency  of  collecting  the  data, 
and  who  is  in  charge  of  the  collection  (Greiner,  1996).  The  analysis,  or  throughput,  is 
influenced  by  the  techniques  used  to  transform  raw  data  into  measures  of  performance. 
The  nature  of  the  results  and  how  those  results  are  communicated  to  users  within  an 
organization  influence  output  (Greiner,  1996).  Greiner  concludes  that  the  measurement 
of  government  performance  requires  the  collection  and  reporting  of  data  at  each  point 
along  the  causal  chain — inputs,  throughput  (processes),  outputs,  and  outcomes  (Greiner, 
1996). 

Depending  on  the  organization's  objectives,  agencies  may  have  to  use  more  than 
one  kind  of  measurement  of  performance.  Indeed,  the  multidimensional  nature  of  many 
public  sector  activities  demands  the  use  of  multiple  performance  measures.  Thus,  clearly 
identified  agency  objectives  are  central  to  the  development  of  any  performance 
measurement  system. 

2.  Differentiating  Between  Process  and  Results-Oriented  Systems 

The  literature  distinguishes  between  measurement  systems  that  focus  on  results 
from  those  that  focus  on  process.  If  desired  results  is  the  target,  then  process 
specifications  are  derived  from  the  desired  results  (Halachmi  and  Bouckaert,  1996).  With 
this  focus,  the  purpose  of  the  measurement  system  is  to  assess  the  attainment  of  the 
desired  results,  or  outcomes,  based  on  clearly  defined  agency  objectives.  Outcome 
measurement  can  be  used  in  a  retrospective  role—to  determine  whether  the  expected 
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benefits  of  a  public  sector  program  have  materialized  (Smith,  1996).  It  can  also  be  used 
in  a  prospective  role— to  guide  future  resource  allocation  decisions.  A  results-oriented 
system,  therefore,  attempts  to  measure  the  outcomes—or  effectiveness— of  a  given 
program,  and  is  the  focus  of  this  thesis. 

A  process-focused  performance  measurement  system  assesses  efforts  to 
reengineer  agency  processes  in  order  to  achieve  greater  efficiency.  The  focus  is  on  the 
causal  chain  from  inputs  to  outputs.  Process-focused  systems  measure  the  productivity  of 
inputs  used  to  create  outputs  in  various  program  processes.  It  is  a  ''clinical  approach"  to 
management;  one  that  is  more  concerned  about  the  procedural  aspects  of  the  operation 
than  the  results  of  the  operation  (Halachmi  and  Bouckaert.  1996,  p.  4). 

3.  Purpose  of  Performance  Measurement 

Hatry  states  that  the  "ultimate  purpose  of  performance  measurements  is  to  use  the 
measurement  information  to  help  make  improvements — whether  to  expand,  delete,  or 
modify  programs"  (Hatry,  1996,  p.  viiii).  Greiner  lists  several  practical  reasons  for 
implementing  performance  measurement:  improvement  of  performance,  of  motivation,  of 
communications,  and  of  accountability  (Greiner,  1996). 

Improving  performance  is  the  most  important  application  of  performance 
measurement.  "Performance  measures  can  be  used  to  enhance  planning,  budgeting,  and 
service  delivery  by  providing  timely  feedback  of  results"  (Greiner,  1996.  p.  19).  The 
availability  of  performance  information  makes  continuous  reassessment  and  improvement 
possible.  Enhancing  motivation  is  possible  when  performance  information  is  linked  to 
incentives    to    stimulate    employees.        Performance     information     also     encourages 
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communication  between  managers,  government  officials,  and  elected  representatives 
(Greiner,  1996).  Finally,  performance  data  can  be  used  to  show  various  interest  groups 
just  what  they  are  getting  for  their  tax  dollars.  Indeed,  regular  performance  reports  can  be 
viewed  "as  a  way  to  keep  the  public  involved  in  the  process  of  government"  (Greiner, 
1996,  p.  20). 

An  increasing  number  of  critics  have  recognized  that  the  federal  government  has 
developed  two  defects  that  are  central  to  its  existence:  (a)  it  does  not  know  how  to  tell 
whether  many  of  the  things  it  does  are  worth  doing  at  all;  and  (b)  when  it  does  decide  that 
something  is  worth  doing,  it  does  not  know  how  to  create  and  sustain  a  program  capable 
of  achieving  the  results  it  seeks  (Quade,  1984).  In  other  words,  what  government  lacks  is 
some  sort  of  regulator  that  tells  it  when  a  program  has  ceased  to  be  effective  or  could  be 
made  more  efficient.  With  increasing  emphasis  on  providing  more  value  for  money  with 
limited  resources,  an  assessment  of  performance  is  needed  to  inform  the  decision-making 
process  and  may  ultimately  become  a  matter  of  government  survival  (Greiner,  1996). 

Many  authors  agree  that  information  and  knowledge,  not  land,  labor,  and  capital, 
are  the  key  resources  in  a  "knowledge  society."  Managing  change  and  ensuring  that 
informed  decisions  are  made  quickly  requires  timely  and  complete  information  on 
performance.  "Without  adequate  measurement,  managers  and  employees  within  the 
agency,  other  agencies  and  oversight  bodies,  interest  groups,  the  public,  or  the  courts  may 
not  be  able  to  document  changes  in  performance"  (Halachmi  and  Bouckaert,  1996,  p.  3). 


14 


4.  Who  Needs  the  Information  Provided  by  Performance  Measurement 

The  public  management  literature  indicates  that  performance  data  is  useful  to 
many  parties.  Program  managers  need  regular  project  performance  information  to 
identify  where  their  programs  are  doing  well,  where  not  well,  and  to  help  identify 
improvement  actions  (Halachmi  and  Bouckaert.  1996).  Staff  needs  this  information  to 
determine  how  well  they  are  doing.  The  media  and  the  public  also  need  performance 
information  in  order  to  fully  participate  in  a  democratic  government. 

In  government,  performance  measurement  can  be  used  by  the  legislature  and 
within  the  staff  function  of  various  government  agencies.  Halachmi  and  Bouckaert  write: 

It  can  be  a  budget  decision  tool,  determining  where  to  allocate  scarce 
funds  to  achieve  the  greatest  impact.  It  can  be  used  by  senior  managers  to 
fine-tune  the  agency  through  corrective  measures  or  to  provide  front-line 
employees  with  feedback  about  their  performance.  It  can  provide 
information  for  planning  by  suggesting  which  operations  need  to  be  re- 
engineered  to  be  more  effective  and  efficient.  It  can  be  a  communications 
tool,  informing  clients  of  the  level  of  service  achieved.  For  clients,  it  can 
be  a  feedback  tool,  allowing  them  an  opportunity  to  indicate  how  services 
meet  or  do  not  meet  their  needs  (Halachmi  and  Bouckaert,  1996.  p.  6). 

The  literature  also  emphasizes  that  all  parties  must  understand  the  limitations  as  to 
what  these  data  tell  and  what  they  do  not  tell,  as  performance  measurement  systems  have 
many  potential  problems. 

5.  Current  Public  Sector  Utilization 

A  1992  GAO  survey  of  102  federal  agencies  found  that  all  of  them  collected  a 
wide  variety  of  performance  measures  (Greiner.  1996).  However,  according  to  Greiner. 


15 


Such  statistics  often  give  a  false  impression  of  the  scope  and  depth  of  the 
performance  measurement  efforts  underway.  The  reported  measurement 
activities  often  focus  primarily  on  input  and  workload  measures,  and  in 
many  instances  they  involve  only  limited  number  of  agencies  and 
applications  (Greiner,  1996,  p.  14). 

The  1992  GAO  report  agreed,  stating  that,  although  most  agencies  employed  a  variety  of 
performance  measures,  substantive  use  of  such  information  was  limited.  The  GAO 
survey  concluded  that  many  federal  agencies  only  collected  outcome  data  for 
informational  purposes  and  few  used  performance  measures  to  facilitate  the  achievement 
of  long-term  objectives  (Greiner,  1996). 

Other  authors  have  arrived  at  similar  conclusions.  Wholey  and  Hatry  observe  that 
"most  reports  on  government  performance  still  focus  on  resource  expenditures  and  the 
quantity  of  services  delivered.:.;  few  government  agencies  provide  timely  information  on 
the  quality  and  outcomes  of  their  major  programs"  (Wholey  and  Hatry,  1992,  p.  604). 
Cave,  Kogan,  and  Smith  add  that  in  the  public  sector  there  is  often  "a  need  for  greater 
attention  to  quality  of  output  and  customer  service,  and  levels  of  service  quality  cannot  be 
adequately  proxied  by  physical  output  measures"  (Cave,  Kogan,  and  Smith,  1990,  p.  10). 

The  performance  evaluation  literature  concludes  that  current  federal  agencies  have 
underutilized  existing  measurement  systems.  While  performance  data  exists,  agencies 
fail  to  use  the  information  to  motivate  long-term  mission  accomplishment  (Wholey  and 
Hatry,  1992). 
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6.  Problems  Associated  with  Public  Sector  Performance  Evaluation 

Understanding  the  lack  of  public  sector  performance  evaluation  requires  an 
examination  of  the  institutional,  technical,  and  financial  obstacles  associated  with  any 
measurement  system.  These  "types  of  obstacles  have  hampered  (and  continue  to  hamper) 
wider  acceptance  and  application  of  performance  measurement  in  the  public  sector*' 
(Greiner,  1996,  p.  16). 

a.  Institutional  Obstacles 

Several  common  characteristics  of  American  government  contribute  to  the 
limited  use  of  performance  measurement  in  the  public  sector.  First,  political  pressures 
often  outweigh  the  impact  of  data  in  the  bureaucratic  decision-making  process  (Greiner, 
1996).  All  too  often  performance  measurement  efforts  carry  less  influence  than  the 
actions  of  powerful  elected  officials,  interest  groups,  and  agency  officials. 

The  presence  of  multiple  self-interested  constituencies  also  makes  it 
difficult — if  not  impossible — "to  reach  agreement  on  service  goals  and  objectives  that 
must  proceed  performance  measuremenf  (Greiner,  1996,  p.  21).  The  impact  of  multiple 
constituencies  on  public  policy  receives  extensive  treatment  in  the  performance 
evaluation  literature.  For  example,  Hitch  recognized  that: 

National  objectives  can  only  be  some  combination  or  distillation  of  the 
objectives  of  people  who  comprise  (or  rule)  the  nation;  and  we  should 
learn  to  be  skeptical  and  critical  of  the  verbalizations  and  rationalizations 
that  pass  for  national  objectives  (Hitch.  1960). 

A  1993  Congressional  Budget  Office  (CBO)  report  concluded  that,  "if  a 
program's   objectives  cannot  be   determined,   performance   measures   will   always   be 
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ambiguous,  if  not  superfluous"  (CBO,  1993,  p.  5).  Unlike  the  corporate  model  with  a 
bottom  line,  differing  objectives  of  multiple  constituencies  cannot  all  be  measured  via  a 
single  performance  index  (Boschken,  1994).  In  a  plural  society  diverse  constituencies 
want  different  things  from  their  public  services.  They  demand  performance  information 
concerning  multiple,  and  often  conflicting,  objectives.  Implementing  a  performance 
measurement  system  that  satisfies  each  constituency  is  often  impossible  and  represents  a 
major  obstacle  to  public  sector  performance  evaluation. 

Utilization  of  performance  measures  has  also  been  unfamiliar  to  the 
management  approach  of  many  public  sector  managers.  These  managers  are  often 
unaccustomed  to  using  quantitative  data  to  aid  decision-making.  A  related  institutional 
obstacle  to  the  acceptance  of  performance  measurement  is  a  government's  "political  self- 
confidence"  (Greiner,  1996,  p.  17).  Are  administrators  and  managers  willing  to  accept 
both  bad  and  good  reports?  "The  prospect  of  accurate  performance  information  and 
increased  accountability  can  be  very  threatening  to  some  elected  officials,  managers,  line 
personnel,  and  interest  groups"  (Greiner,  1996,  p.  17). 

These  factors  combine  to  form  a  lack  of  institutional  readiness  for 
performance  evaluation. 

b.  Technical  Obstacles 

Numerous  technical  obstacles  inhibit  performance  evaluation.  Like 
institutional  obstacles,  these  technical  challenges  represent  significant  impediments  to  the 
measurement  process. 
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At  the  most  basic  level  is  the  lack  of  a  common  definition  of 
productivity — a  key  element  of  performance  measurement.  Greiner  points  out  that  no 
consensus  exists  on  the  definition  of  government  productivity,  let  alone  how  it  is  to  be 
measured  (Greiner,  1996).  Both  measures  of  efficiency  and  effectiveness  have 
inappropriately  been  identified  as  definitions  of  productivity. 

Differentiating  between  output  and  outcome  measures  when  measuring 
organizational  productivity  is  a  common  problem.  Output  measurement  traditionally 
refers  to  the  quantity  of  the  good  or  sendee  provided,  and  is  closely  related  to  the  concept 
of  efficiency  (Smith,  1996).  Many  agree  that  there  is  no  shortage  of  quantifiable  output 
indicators.  Largely  dependent  on  output,  outcome  refers  to  "the  impact  on  society  of  a 
particular  public  sector  activity,"  and  is  closely  related  to  the  concept  of  effectiveness 
(Smith,  1996,  p.  1).  Unlike  output,  outcome  is  far  more  difficult  to  measure.  As  Smith 
points  out,  "although  most  public  sector  auditing  bodies  pay  lip-service  to  the  desirability 
of  addressing  effectiveness  issues,  they  have  found  it  difficult  to  put  into  practice  the 
principles  of  outcome  measurement"  (Smith,  1996,  p.  2).  One  reason  that  helps  explain 
the  difficulties  with  outcome  measurement  is  the  fact  that  many  costs  and  benefits  of 
various  programs  are  non-quantifiable.  The  difficulties  of  systematically  measuring  the 
outcomes  of  governmental  programs,  like  security  assistance,  greatly  limit  the  use  of 
performance  measurement  techniques. 

Another  technical  obstacle  to  the  adoption  of  performance  measurement 
systems  is  the  fact  that  government  services  typically  require  several  different  measures 
to  adequately  characterize  their  performance  (Smith.  1996).    The  multifaceted  nature  of 


individual  government  programs  often  means  that  performance  cannot  be  captured  by 
only  one  or  two  measures.  Information  overload,  caused  by  multidimensional  programs, 
often  overwhelms  potential  users  (Smith,  1996). 

Other  important  obstacles  limit  the  adoption  and  use  of  performance 
measurement  systems.  The  lack  of  standards  forjudging  government  performance  limits 
the  utility  of  performance  data.  Agencies  need  a  ready  benchmark  to  measure  their 
performance  data  against.  Without  it  they  may  not  be  motivated  to  corrective  action. 
Making  performance  data  available  in  a  timely  fashion  is  also  difficult.  If  data  is  to  be 
useful,  performance  feedback  must  occur  often  and  be  consistent.  In  reality,  however,  the 
data  collection  process  often  involves  tedious  delays  (Greiner,  1996). 

c.  Financial  Obstacles 

Implementation  of  performance  measurement  systems  can  be  very  costly. 
In  an  era  of  budgetary  constraint  the  availability  of  resources  (staff  time  and  dollars)  for 
performance  evaluation  is  often  limited.  As  Wholey  and  Hatry  point  out,  "the  cost  of 
performance  monitoring  must  always  be  balanced  against  the  value  of  performance 
monitoring  in  improving  government  performance  and  credibility"  (Wholey  and  Hatry, 
1992,  p.  609). 

7.  Summary 

Performance  measurement  in  public  policy  is  complex  and  problematic.  Public 
sector  operations  are  multidimensional,  possess  multiple  or  poorly  defined  objectives, 
lack  comparative  standards,  and  often  result  in  non-quantifiable  outcomes.  Thus,  public 
sector  performance  evaluation  cannot  be  solely  scientific.    It  must  take  into  account  the 
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political  pressures  of  powerful  stakeholders,  the  lack  of  institutional  readiness,  and 
various  technical  obstacles.  Attempts  at  measurement  must  concentrate  not  only  on 
inputs,  but  also  systematically  on  the  entire  value  chain,  including  throughput  and 
outputs.  Public  agencies  must  ultimately  attempt  to  measure  the  effectiveness,  or 
outcomes,  of  their  programs. 

Although  some  areas  continue  to  pose  problems  for  performance  evaluation  in 
government,  many  government  functions  are  yielding  to  performance  measurement 
systems.  A  spectrum  of  approaches  to  performance  evaluation  of  public  programs 
emerges.  Anchoring  one  end  of  the  spectrum  is  the  "technically  rational  paradigm"  that 
treats  performance  measurement  as  an  objective,  scientific  aid  to  decision-making. 
Anchoring  the  opposite  end  of  the  spectrum  is  the  "politically  rational  paradigm"  that 
emphasizes  the  effect  of  political  power  on  the  measurement  of  performance.  Each 
paradigm  employs  various  analytical  techniques  to  quantify  data  and  measure 
performance. 

B.         PERFORMANCE  MEASUREMENT  PARADIGMS 

A  paradigm  is  "a  systematic  statement  of  the  basic  assumptions,  concepts,  and 
propositions  employed  by  a  school  of  analysis"  (Allison,  1971.  p.  32).  Approaches  to 
public  sector  performance  evaluation  form  a  continuum  from  the  "technically  rational 
paradigm"  to  the  "politically  rational  paradigm." 
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1.  The  "Technically  Rational  Paradigm" 

The  most  widely  used  approach  to  public  sector  performance  evaluation  is  the 
technically  rational  paradigm.  Central  to  the  application  of  this  paradigm  is  the 
assumption  that  program  objectives  are  clearly  stated  and  costs  and  resulting  benefits  are 
measurable.  Being  able  to  weigh,  measure,  and  quantify  is  essential  to  this  type  of  policy 
analysis.  Objectives  consensus  is  required  before  any  measures  of  effectiveness  are 
developed.  Its  dominant  appeal  is  that  it  would  discipline  public  choice  so  that  scarce 
public  resources  are  rationally  allocated  to  their  highest  valued  uses  (Byrne,  1987). 
Examples  of  well-documented  technically  rational  techniques  include  economic  cost- 
benefit  analysis,  performance  ratios,  and  the  linear  programming  model.  This  paradigm 
also  includes  traditional  operations  research  techniques  that  address  the  problem  of 
multidimensionality  such  as  data  envelopment  analysis  and  multiattribute  utility  theory 
(Greiner,  1996). 

Users  of  technically  rational  techniques,  like  cost-benefit  analysis,  within 
government  fall  generally  into  two  groups;  those  who  recognize  decisions  as  necessarily 
imperfect  and  treat  techniques  like  cost-benefit  analysis  as  an  organizing  rather  than  a 
calculative  framework  for  addressing  problems,  and  those  which  see  the  technique  as  a 
way  to  establish  formal  criteria  in  government  decision  making  (Byrne,  1987).  Thus,  a 
split  emerges  between  those  who  use  technically  rational  techniques  as  an  informative  aid 
to  decision-making,  and  those  who  advocate  their  rigorous  application  in  government. 
Critical  review,  however,  suggests  that  the  rigorous  application  of  techniques  like  cost- 
benefit  analysis  has  generated  numerous  problems. 
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One  obvious  problem  is  quantifying  intangibles  like  quality  of  life  or  the  value  of 
a  human  life.  As  stated  earlier,  to  proceed  logically  in  this  paradigm  one  must  be  able  to 
weigh,  measure,  and  quantify.  There  is  an  inherent  bias  to  favor  elements  that  are 
quantifiable — even  if  they  are  not  relevant  to  the  critical  questions  being  asked.  This 
shortcoming  surfaces  in  the  area  of  public  policy.  Because  many  public  programs 
produce  benefits  that  are  not  quantifiable,  only  their  dollar  costs  are  readily  calculated. 
The  result  is  an  "impression  of  accuracy  that  does  not  exist  but  also  the  notion  that  solely 
that  which  is  quantifiable  is  significant"  (Hoos,  1972,  p.  139).  Applying  economic  cost- 
benefit  analysis  to  public  sector  decisions  risks  the  "distortion,  and  ultimately,  superficial 
treatment  of  matters  of  the  deepest  consequence"'  (Hoos,  1972.  p.  xxix).  The  inevitable 
consequence  of  this  urge  to  scientific  objectivity  is  the  tendency  to  treat  the  matter  under 
study  as  though  it  was  primarily  economic  and  thus  can  be  judged  through  technically 
rational  analysis. 

The  existence  of  numerous  political  constituencies  raises  the  issue  of  multiple 
objectives.  Application  of  the  technically  rational  paradigm  assumes  a  consensus  on 
objectives.  In  order  to  technically  measure  performance  one  must  have  a  clear  idea  what 
it  is  that  they  are  measuring,  and  to  what  their  results  will  be  compared.  Getting 
competing  constituencies  to  agree  on  a  single  set  of  objectives  is  often  difficult.  But 
agreement  on  a  single  set  of  objectives  is  only  half  of  the  problem.  What.  then,  should 
the  criteria  be  for  selecting  one  alternative  over  another  and  for  deciding  whether 
performance  has  met  the  stated  objectives?  Assuming  that  quantifiable  data  exists, 
simply  calculating  a  performance  ratio  does  not  yield  the  ultimate  answer.   The  reality  of 
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public  sector  policy  analysis  is  that  political  pressures  play  a  key  role  in  the  setting  of 
objectives  and  the  selection  of  performance  criteria. 

The  true  value,  and  main  difficulty,  of  technically  rational  methods  such  as  cost- 
benefit  analysis  is  its  forced  concentration  on  the  measurable  elements  rather  than  the 
whole.  Through  this  "piece-meal  fragmentation"  or  "sub-optimization,"  analysts  juggle 
the  parts  that  they  can  handle  (Hoos,  1972,  p.  138).  While  measures  of  overall 
effectiveness  remain  difficult  to  quantify,  and  thus  outside  the  realm  of  rigorous  cost- 
benefit  analysis,  pieces  of  the  overall  puzzle  may  be  within  grasp.  The  goal,  then,  ought 
not  to  be  a  definitive  solution  but  a  reasonable  one,  founded  on  an  informed  under- 
standing of  the  measurable  sub-elements  of  the  larger  objective. 

2.  The  "Politically  Rational  Paradigm" 

The  Founding  Fathers'  concerns  for  liberty  resulted  in  a  tripartite  division  of 
powers  within  the  national  government.  This  deliberately  pluralist  formulation  is  the 
source  of  many  of  the  difficulties  surrounding  performance  measurement  in  the  public 
sector.  The  assessment  of  effectiveness  requires  specification  of  objectives  against  which 
outcomes  can  be  measured.  But  whose  objectives  should  be  considered?  Technically 
rational  performance  measurement  models  have  "a  tradition  of  defining  'performance' 
from  the  perspective  of  one  stakeholder  in  particular"  (Bovaird,  1996,  p.  146).  As  a 
result,  technically  rational  measurement  systems  are  often  totally  at  odds  with  their 
political  environment. 
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a.  Multiple  Objectives 

Public  agencies  that  are  owned  and  funded  by  democratic  governments 
operate  under  political  authority  (Rainey,  1991).  Multiple  constituencies  (stakeholders), 
such  as  interest  groups,  elected  and  agency  officials,  the  media,  legislative  bodies,  and  the 
general  public,  heavily  influence  the  exercise  of  that  authority.  The  presence  of  multiple 
constituencies  within  the  political  system  results  in  the  pursuit  of  multiple,  intangible,  and 
often,  conflicting,  objectives  (Rainey,  1991).  As  a  result,  performance  evaluation 
becomes  highly  subjective. 

To  reach  consensus  on  objectives,  against  which  outcomes  should  be 
measured,  all  groups  affected  by  the  policy  must  share  a  common  interest  that  outweighs 
their  particular  interests.  The  relative  political  power  of  each  group,  however,  determines 
their  impact  on  the  objectives-setting  process.  Objectives  consensus,  therefore,  is  not 
objective  or  equally  weighted,  but  rather  a  result  of  the  power  and  bargaining  skill  among 
various  stakeholders  (Allison,  1971).  A  single  set  of  objectives  "results  from 
compromise,  conflict,  and  confusion  [among]  officials  with  diverse  interests  and  unequal 
influence"' (Allison,  1971,  p.  162).  As  Smith  points  out. 

It  is  rare  to  find  consensus  about  what  constitutes  the  objectives  of  a  public 
sector  program.  Thus,  in  making  a  choice  of  outcome  measures,  the 
outcome  measurer  is  implicitly  reflecting  the  objectives  of  one  group  of 
stakeholders,  possibly  at  the  expense  of  another  group.  That  is.  no 
outcome  measurement  scheme  can  be  free  of  values,  and  any  evaluation  of 
outcome  must  be  in  respect  of  a  particular  set  of  objectives  (Smith.  1996. 
P- 9). 
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Bovaird  agrees,  stating, 

"Performance"  is  not  a  unitary  concept,  within  an  unambiguous  meaning. 
Rather,  it  must  be  viewed  as  a  set  of  information  about  achievements  of 
varying  significance  to  differing  stakeholders.  Performance  measures 
must  always  be  viewed  within  a  multidimensional  framework,  and  should 
be  expected  to  be  multidimensional  in  their  form  and  content.  If  they 
appear  one-dimensional,  it  is  likely  to  be  because  they  are  of  limited 
relevance.  In  practice,  no  performance  information  is  other  than  partial, 
imperfect,  and  indicative  (Bovaird,  1996,  p.  147). 

A  lack  of  consensus  on  program  objectives  is  a  common  reality-  of  the 
politically  rational  paradigm.  When  consensus  does  exist,  it  is  a  result  of  the  political 
interaction  among  powerful  stakeholders.  The  value  of  the  paradigm  lies  in  its 
recognition  that  public  sector  performance  evaluation  must  first  identify  key  stakeholders 
and  their  respective  objectives  before  applying  a  technically  rational  measurement 
technique.  The  process  of  public  sector  performance  measurement  is  thus  broken-up  into 
four  stages: 

1 .  Identifying  Stakeholders; 

2.  Identifying  Objectives; 

3.  Developing  Performance  Indices; 

4.  Applying  technical  rational  performance  measurement  techniques. 

b.  Identifying  Stakeholders,  Objectives,  and  Indices:  The  Objectives 

Hierarchy  Approach 

A  useful  approach  to  systematically  link  key  stakeholders  and  their 

respective  objectives  to  performance  indicators  is  the  Objectives  Hierarchy.4      An 


4  The  Objectives  Hierarchy  is  the  first  step  in  Thomas  Saaty's  Analytical  Hierarchy  Process  (AHP).  While 
the  researcher  will  use  the  Objectives  Hierarchy  approach  throughout  this  study  to  break  broad  objectives 
down  into  sub-objectives  and  specific  attributes,  the  remaining  AHP  techniques  will  not  be  applied. 
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Objectives  Hierarchy  stratifies  into  levels  the  relevant  stakeholders  and  their  respective 
objectives.  A  Hierarchy  can  be  used  to  work  down  from  each  agency's  high  level 
objective  (e.g.,  overall  organizational  performance)  to  relevant  sub-objectives,  to  specific 
attributes  that  characterize  those  sub-objectives,  and  finally  to  measurable  characteristics 
that  allows  quantification  of  the  desired  attributes  (Melese.  Lowe,  and  Stroup.  1997).  A 
simple  example  of  an  objectives  hierarchy  for  the  Department  of  State  is  illustrated  in 
Figure  1. 
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Figure  1.  Example  of  an  Objectives  Hierarchy 
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In  practice  there  is  no  set  procedure  for  generating  the  objectives,  sub- 
objectives,  and  specific  attributes  to  be  included  in  a  hierarchy  (Saaty,  1980).  It  is 
important  to  remember,  however,  that  the  ultimate  organizational  goal  (objective)  needs 
to  be  identified  and  placed  at  the  top  of  the  hierarchy. 

c.  Summary 

Disagreement  on  evaluative  frameworks  exists  not  only  because  of 
technical  obstacles,  but  also  as  a  consequence  of  underlying  political  divisions  (Fainstein, 
1987).  As  a  result,  performance  measurement  can  never  be  entirely  scientific.  Decision- 
making is  much  less  systematic  and  rational  than  commonly  supposed.  As  an  aid  to 
decision-making,  performance  evaluation  is  far  less  objective  than  the  technically  rational 
paradigm  presumes.  An  understanding  of  the  "governmental  machine"'  of  organizational 
processes  and  political  actors  is  required  before  a  measurement  system  can  be  developed 
(Allison,  1971).  It  is  imperative  to  identify  the  stakeholders,  and  to  what  extent 
objectives  consensus  exists,  prior  to  adopting  any  technically  rational  performance  index. 
The  Objectives  Hierarchy  process  is  a  useful  approach  to  analyzing  multiple  agencies  and 
their  objectives. 

C.         CHAPTER  SUMMARY 

Public  sector  performance  evaluation  is  complex  and  controversial.  Measurement 
of  program  efficiency  involves  collecting  and  reporting  on  input,  throughput,  and  output 
factors  in  an  effort  to  improve  agency  processes.  Assessing  the  effectiveness  of  a 
program  requires  measuring  outcomes  against  stated  objectives.    Various  government 
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reports  and  private  scholars  have  remarked  on  the  lack  of  results-oriented  measurement  in 
government.  Many  reasons  have  been  proposed  for  this,  including  institutional, 
technical,  and  financial  obstacles.  Many  practical  reasons  exist,  however,  for 
implementing  performance  measurement  systems  in  government. 

A  spectrum  of  approaches  to  public  sector  performance  evaluation  exists  within 
the  literature.  Anchoring  one  end  of  the  spectrum  is  the  ''technically  rational  paradigm" 
that  presupposes  objectives  consensus  and  treats  performance  measurement  as  a  coldly 
objective  and  scientific  aid  to  decision-making.  On  the  other  end  is  the  "politically 
rational  paradigm1'  that  recognizes  that  objectives  consensus,  when  it  exists,  is  a  result  of 
political  bargaining  among  multiple  stakeholders.  It  emphasizes  identifying  stakeholders 
and  their  respective  objectives  as  a  preliminary  step  in  the  performance  measurement 
process. 
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III.   METHODOLOGY 

A.  INTRODUCTION 

This  chapter  will  outline  the  methodology  used  to  conduct  this  study.  It  will 
describe  the  selected  research  focus,  the  primary  research  methods  used,  and  how  the 
public  sector  performance  evaluation  process  outlined  in  Chapter  II  is  linked  to  the  data 
collection  and  analysis  stages  of  the  study.  The  chapter  will  conclude  with  a  detailed 
description  of  how  each  step  in  the  public  sector  performance  evaluation  process  w:as 
accomplished. 

B.  RESEARCH  FOCUS 

The  objective  of  this  thesis  is  to  address  how  the  Defense  Security  Cooperation 
Agency  (DSCA)  and  the  Departments  of  State  and  Defense  can  tackle  the  challenge  of 
measuring  the  effectiveness  and  performance  of  their  security  assistance  programs.  The 
researcher  chose  to  focus  on  the  Expanded  International  Military  Education  and  Training 
(E-IMET)  military-to-civilian  security  assistance  program  due  to  the  inherent  subjective 
nature  of  international  assistance  efforts.  A  commonly  held  view  within  the  security 
assistance  community  is  that  the  effectiveness  of  security  assistance  programs  cannot  be 
measured  with  certainty  due  a  lack  of  quantitative  data,  thereby  making  strict  compliance 
with  the  1993  Government  Performance  and  Results  Act  impossible.  It  is  the 
researcher's  hypothesis,  however,  that  objectively  measuring  the  effectiveness  of  security 
assistance  programs  is  possible.    In  order  to  place  constructive  limitations  on  the  study. 
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and  because  it  is  the  potential  core  of  an  evolving  military-to-civilian  security  assistance 
effort,  the  researcher  focused  entirely  on  the  E-IMET  program. 

In  the  spirit  of  the  Results  Act,  any  government  agency  and  program  could  have 
been  selected.  DSC  A  was  chosen  as  the  central  point  of  contact  because  it  is  organic  to 
the  Defense  Department,  is  responsible  for  the  implementation  of  the  E-IMET  program, 
and  is  faced  with  the  same  challenges  as  more  traditional  State  Department  security 
assistance  organizations  (i.e.,:  The  Agency  for  International  Development  (USAID)). 

C.         RESEARCH  METHODS 

This  study  makes  use  of  two  common  public  administration  research  methods. 
The  first  is  the  collection  of  data  through  unobtrusive  measures,  specifically  archival 
research,  and  the  second  is  the  collection  and  verification  of  data  through  specialized 
interviewing. 

1.  Unobtrusive  Measures:  Public  Archival  Data 

Webb  et  al.  (1966)  point  out  that  archival  sources  of  information  have  been 
widely  used  in  public  administration  research  for  many  years  (Webb,  Campbell, 
Schwartz,  and  Sechrest,  1966).  Archival  sources  of  data  are  written  materials  that  are 
available  from  public  and  private  records  (Yaeger,  1989).  Public  archives  include 
"ongoing  records  of  society,  such  as  actuarial  records,  voting  records,  government 
budgets,  and  communications  media,  which  are  produced  and  paid  for  by  someone  other 
than  the  researcher"  (Yaeger,  1989,  p.  714).  The  archival  data  collected  in  this  study 
come  solely  from  public  records. 
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Archival  sources  are  further  classified  as  primary  or  secondary.  Primary  public 
sources  used  in  this  study  include  speeches,  congressional  testimony,  reports,  and 
documents  published  by  various  U.S.  government  agencies.  Examples  include  reports  by 
the  General  Accounting  Office  (GAO)  and  agency-specific  publications  such  as  the 
National  Military  Strategy  (DoD).  Data  collected  from  these  primary  public  sources 
include  formal  organizational  charts,  missions,  and  objectives. 

Secondary  public  sources  include  the  published  books,  articles,  studies,  papers, 
and  lectures  of  academics  (Yaeger,  1989).  Secondary  sources  are  used  in  this  study  to 
inform  the  performance  measurement  process  and  provide  the  background  information 
necessary  to  critically  analyze  the  data  collected  from  primary  public  sources.  .An 
example  of  a  secondary  public  source  is  the  Public  Administration  Review,  the  oldest  of 
the  public  administration  journals  (Yaeger,  1989).  Data  collected  from  these  sources 
helped  form  the  continuum  of  performance  evaluation  paradigms  described  in  Chapter  II 
as  well  as  the  public  sector  performance  measurement  process  used  throughout  this 
thesis.  Secondary  archival  data  also  provided  information  critical  to  the  development  of 
the  E-IMET  performance  indices  found  in  Chapter  VI. 

Archival  sources  are  not  without  problems,  however.  First,  archives  may  contain 
errors  or  be  incomplete  (Yaeger,  1989).  Second,  some  archival  materials  "suffer  from 
from  selective  deposit"  (Yaeger,  1989,  p.  720).  Another  common  problem  is  that  of 
selective  survivability.  This  may  happen  with  public  sources  when  a  political 
administration  or  agency  removes  records  that  reflect  negatively  upon  the  organization 
(Yaeger.  1989).  While  these  problems  might  exist,  the  researcher  took  steps  to  limit  their 
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effects.  First,  documents  were  checked  for  internal  consistency.  More  importantly, 
multiple  sources  were  used  to  ensure  a  fair  picture  emerged  of  a  given  issue.  Agency- 
specific  sources  were  compared  to  reports  from  other  agencies,  like  the  GAO.  or  to 
similar  secondary  sources.  As  will  be  described  in  the  next  section,  key  archival  data 
were  also  verified  through  the  specialized  interview  process. 

While  there  is  no  complete  assurance  that  the  data  offer  a  comprehensive 
representation  of  all  the  stakeholders,  the  researcher  made  every  effort  to  incorporate 
archival  data  from  all  parties. 

2.  Specialized  Interviews 

Interviews  are  frequently  used  to  collect  data  and  have  been  used  throughout  the 
history  of  public  administration  research  (Yaeger,  1989).  One  type  of  interview,  known 
as  the  elite  or  "specialized  interview,"  is  used  to  collect  data  from  a  limited  group  of 
individuals  or  "key  informants"  (Yaeger,  1989,  p.  698).  Key  informants  are  "individuals, 
who  because  of  their  position,  are  likely  to  have  been  involved  either  in  making  a 
decision  of  interest  or  in  carrying  it  out"  (Yaeger,  1989,  p.  698).  The  special  knowledge 
and  experience  that  they  have  may  enable  them  to  comment  on  or  verify  the  accuracy  of 
the  information  gained  during  the  course  of  the  research  project  (Yaeger,  1989).  With 
specialized  interviews  the  number  of  subjects  usually  is  limited.  Rather  than  interviewing 
many  subjects,  key  informants  are  selected  because  of  the  position  they  hold  or  because 
of  their  experiences. 

In  August  1998  the  researcher  traveled  to  Washington  D.C.  to  interview  various 
key  E-IMET  informants.   An  example  of  a  key  informant  used  in  this  study  is  Ms.  Rita 
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Verry.  head  of  the  E-IMET  program  for  DSCA.  Specialized  interviews  with  other 
selected  officials  within  the  E-IMET  community  were  used  to  verify  and  help  interpret 
various  archival  data.  Data  gathered  during  specialized  interviews  with  various  academic 
professionals  at  the  Naval  Postgraduate  School  (NPS),  the  Defense  Resources 
Management  Institute  (DRMI),  the  International  Health  Resources  Management  (IHRM) 
Program,  the  Center  for  Civil-Military  Relations  (CCMR),  and  the  Defense  Institute  of 
International  Legal  Studies  (DHLS)  aided  in  the  development  of  performance  indicators 
for  the  E-IMET  program. 

Various  specialized  interview  formats  exist.  The  one  used  in  this  study  was  the 
nonscheduled  or  nondirective  interview.  In  this  type  of  specialized  interview  the 
questions  asked  of  each  subject  are  not  necessarily  the  same  and  may  "differ  in 
significant  ways  depending  on  the  different  positions  and  experiences  of  each  subject" 
(Yaeger,  1989,  p.  699).  No  preconceived  set  of  questions  was  developed  in  advance  in 
order  to  encourage  the  respondents  to  describe  their  experiences  and  interpretations  as 
freely  as  possible.  The  questions  posed  during  this  study  were  often  open-ended  and 
intended  to  get  the  subject  to  talk  in  his  or  her  own  terms  about  the  E-IMET  program. 
The  data  collected  through  these  interviews  took  shape  as  the  process  went  along,  with 
tentative  generalizations  being  tested  in  subsequent  interviews  of  other  subjects. 

It  is  important  to  note  that  the  key  to  conducting  a  specialized  nondirective 
interview  is  sound  background  information.  This  is  where  the  unobtrusive  archival  and 
the  specialized  interview  research  methods  intersect  to  increase  the  adequacy,  reliability, 
and  representativeness  of  the  data  collected.  Data  gathered  from  archival  material  served 
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as  the  basis  for  the  researcher's  questions  during  the  interviews.  The  interviews  served  as 
a  way  of  verifying  the  researcher's  generalizations  concerning  performance  evaluation  of 
the  E-IMET  program. 

Like  archival  data,  interview  data  are  also  not  without  errors.  Potential  problems 
include  omission  and  underreporting,  telescoping  or  overreporting,  and  lack  of 
communication  skills  (Yaeger,  1989).  These  potential  sources  of  error  were  overcome  by 
the  researcher  through  verification  with  other  data  sources  and  double-checking  with  the 
subject  during  the  interview.  Subsequent  telephone  interviews  were  also  conducted  when 
further  clarification  was  needed. 

D.         LINKING  THE  PUBLIC  SECTOR  PERFORMANCE  MEASUREMENT 
PROCESS  TO  DATA  COLLECTION  AND  ANALYSIS 

The  process  of  public  sector  performance  measurement  used  in  this  thesis  is 
broken  down  into  four  iterative  stages: 

1 .  Identify  the  Key  E-IMET  Stakeholders; 

2.  Identify  the  Objectives  of  U.S.   Security  Assistance  and  the  E-IMET 
Program; 

3.  Develop  Performance  Indices  for  the  E-IMET  Objectives; 

4.  Apply  technically  rational  performance  measurement  techniques. 

The  researcher  will  apply  the  first  three  steps  of  this  process  using  data  gathered 
from  the  two  primary  research  techniques.  Step  #1  consists  of  data  collection  involving 
primary  and  secondary  archival  research.  Step  #2  continues  the  collection  of  archival 
data  but  concludes  with  the  formation  of  a  Global  Objectives  Hierarchy,  thus  moving  the 
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study  into  the  analysis  stage  and  triggering  the  specialized  interview  process.  The 
analysis  stage  will  conclude  with  a  set  of  proposed  performance  indicators  for  each  of  the 
three  E-IMET  program  objectives  and  is  heavily  dependent  on  data  gathered  through 
specialized  interviews  and  secondary  archival  sources.  Step  #4.  actually  measuring  the 
performance  of  the  program,  is  beyond  the  scope  of  this  study.  Conclusions  will  then  be 
drawn  from  the  experiences  and  insight  gained  from  the  application  of  the  process. 

E.         THE  PERFORMANCE  MEASUREMENT  PROCESS  AND  E-IMET 
1.  Step  #1 :  Identifying  the  Key  E-IMET  Stakeholders 

The  purpose  of  this  step  is  to  identify  and  classify  the  key  stakeholders  of  the  E- 
IMET  program.  This  process  will  take  place  in  five  sequential  steps. 

First,  the  concept  of  a  stakeholder  will  be  clearly  defined  using  secondary  archival 
sources.  Internal  and  external  E-IMET  stakeholders  will  then  be  identified  using  primary 
archival  sources  including  security  assistance  doctrinal  publications.  Once  individual 
stakeholders  have  been  identified,  they  will  be  grouped  according  to  their  parent 
governmental  agencies  and  the  stakes  they  share  in  the  evaluation  of  E-IMET 
performance.  For  example,  DSCA,  which  is  identified  as  a  key  individual  stakeholder,  is 
grouped  under  its  parent  agency,  the  Department  of  Defense.  The  assumption  here  is  that 
individual  stakeholders  within  the  same  agency  share  common  views  on  E-IMET 
performance  evaluation  and  have   similar  effects   on  the   E-IMET  program.5      This 


5  It  should  be  noted  that  it  is  entirely  possible  that  individual  stakeholders  could  have  different  views  and 
effects  on  a  given  program  than  their  respective  agency  as  a  whole.  Within  a  government  agency, 
however,  it  is  reasonable  to  assume  that  the  parent  organization  (DoD)  would,  to  a  certain  extent,  govern 
the  effects  of  their  organic  individual  agencies  (DSCA),  thus  making  a  generalization  (grouping)  possible. 


assumption  is  made  to  simplify  the  analysis  in  order  to  remain  within  the  time  constraints 

of  the  study.    The  basis  for  this  grouping  will  be  the  organizational  charts  found  in  the 

primary  archival  sources.  Each  group's  stake  in  E-IMET  program  will  then  be  discussed 

in  order  to  better  understand  the  effects  that  they  have  on  the  program's  overall 

objectives.     This  is  a  very  subjective  step  and  will  be  informed  by  agency-specific 

documents  and  verified  by  subsequent  specialized  interviews.     The  last  step  in  the 

stakeholder  identification  process  is  to  formally  organize  the  various  stakeholder  groups 

into  a  "stakeholder  map"  which  will  graphically  present  the  potentially  conflicting  effects 

of  the  various  stakeholders  on  E-IMET  objectives  consensus.    This,  too,  is  a  subjective 

step,  but  will  be  informed  by  agency-specific  documents  and  verified  by  subsequent 

specialized  interviews. 

2.  Step  #2:    Identifying  the  Objectives  of  U.S.  Security  Assistance  and 

the  E-IMET  Program 

The  purpose  of  this  step  is  to  construct  a  broad  Global  Security  Assistance 
Objectives  Hierarchy  that  will  serve  as  a  basis  for  objective  performance  evaluation  and 
link  the  three  E-IMET  objectives  back  to  each  of  the  Executive  Branch  stakeholders.6 
This  process  will  take  place  in  five  sequential  steps. 

First,  each  stakeholder  group's  security  assistance  objectives  will  be  outlined 
using  official  agency-specific  documents.  Once  outlined,  a  subjective  sifting  process  will 


°  Only  Executive  Branch  stakeholder  objectives  were  outlined  in  this  study.  While  Legislative  Branch 
stakeholders  are  identified  and  have  a  profound  effect  on  the  E-IMET  program,  they  do  not  set  specific 
Administration  security  assistance  direction. 
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separate  objectives  from  strategies  (ends  versus  means)  resulting  in  three  separate 
agency-specific  objective  hierarchies.  A  Global  Objectives  Hierarchy  will  then  be 
constructed  using  common  Executive  Branch  stakeholder  objectives.  This  Hierarchy  will 
break  broad  national  objectives  down  into  sub-objectives  and  specific  attributes  that 
characterize  those  sub-objectives  using  the  multiple  objectives  found  in  the  existing 
individual  hierarchies.  While  this  process  is  inherently  subjective  it  is  more  a  function  of 
pattern  recognition  among  the  separate  stakeholder  hierarchies.  The  next  step  is  to 
validate  the  accuracy  of  the  Global  Hierarchy,  as  well  as  the  objectives  selection  criteria 
used  to  construct  it,  through  specialized  interviews  with  key  informants  within  the 
Defense  Department  and  the  State  Department.7  An  opinion  was  also  sought  from 
Legislative  Branch  key  informants  even  though  their  objectives  are  not  specifically 
incorporated  into  the  Hierarchy.  Once  validated,  the  final  step  in  the  objectives 
identification  process  is  to  link  the  specific  E-IMET  objectives  to  the  Global  Hierarchy. 
This  last  step  forms  the  logical  link  between  common  national  security  assistance 
objectives  and  the  E-IMET  program. 

3.  Step  #3:  Developing  Performance  Indices  for  the  E-IMET  Objectives 

The  next  step  in  the  public  sector  performance  evaluation  process,  and  the 
purpose  of  the  last  step  in  this  study,  is  to  propose  indicators  of  performance  for  the  three 
E-IMET  program  objectives.  The  three  main  objectives  of  the  E-IMET  program  are: 


'  Representatives  from  the  Executive  Office  of  the  President  were  unavailable  for  interview.  Their 
national  security  assistance  objectives  are  clearly  defined  in  the  National  Security  Strategy  and  the 
President's  annual  State  of  the  Union  Address,  however. 
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1 .  Respect  for  Civilian  Control  Over  the  Military; 

2.  Improved  Military  Justice  Systems  and  Procedures  In  Accordance  With 
International  Human  Rights; 

3.  Improved  Defense  Resource  Management. 

This  is  the  most  critical  and  problematic  step  in  the  performance  evaluation 
process.  It  involves  identifying  what  qualities  are  inherent  in  each  of  the  three  objectives. 
Quantifiable  indicators  of  performance  must  then  be  matched  to  the  specific  characteristic 
of  the  overall  objective.  In  short,  it  requires  a  deep  understanding  of  civil-military 
relations,  military  justice,  and  resource  management. 

This  step  relies  heavily  on  the  data  received  from  specialized  interviews 
conducted  with  the  academic  professionals  from  the  Naval  Postgraduate  School,  the 
Defense  Resources  Management  Institute,  the  International  Health  Resources 
Management  Program,  the  Center  for  Civil-Military  Relations,  and  Defense  Institute  of 
International  Legal  Studies.  The  experienced  opinions  of  these  professionals,  combined 
with  data  gathered  from  secondary  archival  sources,  will  help  form  individual  objectives 
hierarchies  for  each  E-IMET  objective.  These  hierarchies  will  then  be  used  to  develop 
indicators  of  performance  for  the  E-IMET  program. 

It  is  important  to  note  that  the  result  of  the  process  is  simply  a  starting  point  for 
further  discussion.  The  indicators  found  in  this  study  will  be  developed  largely  outside 
the  influence  of  in-country  security  assistance  officials.  The  researcher  realizes  that  the 
inherently  subjective  nature  of  this  process  must  be  combined  with  informed  country- 
specific  data  in  order  to  yield  the  most  beneficial  measures  of  performance.  The  various 
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Security  Assistance  Officers  (SAOs)  who  serve  as  the  in-country  point  of  contact  for  U.S. 
security  assistance  efforts  were  not  interviewed  for  this  study  due  to  time  and  financial 
constraints.  The  performance  indicators  found  in  Chapter  VI  are  a  result  of  the  critical 
analysis  of  the  three  broad  E-IMET  objectives  and  are  meant  as  a  starting  point  for  the 
future  development  of  country-specific  performance  indicators. 

F.  CHAPTER  SUMMARY 

1.  The  Public  Sector  Performance  Evaluation  Process 

The  four-step  method  to  measure  the  performance  of  public  sector  programs  is  an 
iterative  process  utilizing  data  gathered  through  archival  and  interview  research.  While 
many  steps  in  the  process  are  largely  subjective,  a  concerted  effort  was  made  to 
crosscheck  and  verify  each  assumption.  Primary  archival  data  was  compared  to  similar 
secondary  archival  data  and  specialized  interviews  were  used  to  verify  all  assumptions 
and  generalizations  based  on  archival  material.  While  every  effort  was  made  to  gather 
data  that  is  representative,  adequate,  and  reliable,  limitations  did  exist. 

2.  Limitations  of  the  Study 

The  chief  limitation  of  this  study  is  the  lack  of  past  efforts  to  measure  the 
effectiveness  of  security  assistance  programs.  A  standard  evaluation  format  does  not 
exist  for  security  assistance  programs.  This  created  an  enormous  obstacle  as  the 
researcher  was  forced  to  break  new  ground  concerning  governmental  program  evaluation. 

Another  limitation  placed  on  the  study  was  the  lack  of  access  to  various  E-IMET 
stakeholders.   While  some  stakeholders  were  very  accessible  (DSCA  and  the  Department 


41 


of  State  especially),  many  were  not.  The  face-to-face  interviews  used  in  this  study  were 
conducted  during  a  congressional  recess  making  it  difficult  to  adequately  represent  the 
Legislative  Branch's  position.  Representatives  from  the  Executive  Office  of  the 
President  were  also  unavailable  for  interviews.  Additionally,  various  in-country  U.S. 
Embassy  Security  Assistance  Officers  were  not  contacted  for  their  input  concerning 
region-specific  performance  indicators. 
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IV.        DATA 

A.         INTRODUCTION 

The  purpose  of  the  1993  Government  Performance  and  Results  Act  (GPRA)  is  "to 
improve  Federal  program  effectiveness  and  public  accountability  by  promoting  a  new 
focus  on  results,  service  quality,  and  customer  satisfaction"  (GPRA.  1993.  p.  2).  GPRA 
mandated  that  each  government  agency  submit  to  the  Director  of  the  Office  of 
Management  and  Budget  and  to  Congress  a  strategic  plan  for  program  activities  (GPRA. 
1993).  When  developing  a  strategic  plan,  GPRA  requires  that  the  agency  "consult  with 
Congress,  and  solicit  and  consider  the  views  and  suggestions  of  those  potentially  affected 
by  or  interested  in  such  a  plan"  (GPRA,  1993.  p.  3).  In  short,  GPRA  requires  each 
agency  to  work  with  their  internal  and  external  stakeholders  to  set  agency  direction. 
Thus,  it  is  essential  to  consider  the  various  stakeholders  that  affect  the  agency  or  program 
before  a  set  of  overall  objectives  can  be  developed. 

The  purpose  of  this  chapter  is  to  identify  the  stakeholders  of  the  E-IMET  program. 
Once  the  stakeholders  have  been  identified  they  will  be  grouped  according  to  similar 
stakes  and  organized  into  an  E-IMET  stakeholder  map.  Chapter  V  will  identify  the 
security  assistance  goals  (objectives)  of  each  stakeholder  group  based  on  their  written 
documentation  (i.e.,:  The  National  Security  Strategy)  in  order  to  inform  the  E-IMET 
objectives  setting  process  and  enable  the  development  of  performance  indicators  for  the 
E-IMET  program. 


B.         EXPANDED  IMET  STAKEHOLDERS 
1.  Identification  of  Stakeholders 

Measuring  the  performance  of  the  E-IMET  program  must  begin  with  the 
identification  of  key  stakeholders.  Freeman  defines  a  stakeholder  as  "any  group  or 
individual  who  can  affect  or  is  affected  by  the  achievement  of  an  organization's  purpose" 
(Freeman,  1984).  To  identify  key  stakeholders  Roberts  and  King  ask  a  simple  question: 
"What  are  the  key  stakeholder  groups  that  are  affected  by  or  can  affect  the  policies  and 
programs  which  you  are  attempting  to  introduce?"  (Roberts  and  King,  1989,  p.  68). 

Descriptions  of  a  stakeholder  range  from  the  highly  specific  to  the  general 
(Clarke,  1998).  Stakeholders  can  be  either  internal  or  external  to  the  organization  or 
program  (Roberts  and  King,  1989).  They  can  also  be  potentially  hostile  or  adversarial. 
Any  group,  internal  or  external,  can  potentially  make  a  difference  to  the  organization  or 
program  (Roberts  and  King,  1989).  Once  identified,  individual  stakeholders  are  grouped 
according  to  similar  stakes  and  organized  into  an  overall  "stakeholder  map"  (Roberts  and 
King,  1989). 

The  presence  of  multiple  stakeholder  groups  may  result  in  diverse  or  conflicting 
ideas  concerning  a  program's  objectives.  Thus,  the  careful  identification  of  dominant 
stakeholders  is  the  critical  first  step  in  defining  a  single  set  of  objectives  for  a  program. 
Figure  2  contains  a  list  of  the  dominant  individual  stakeholders  for  the  E-IMET  program. 
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Secretary  of 
The  President  Defense 


The  National  Security 
Council 

Secretary  of  State 

Under  Secretary  for 

International  Security 

Affairs 

Security  Assistance  Program 
Working  Group 

Bureau  of  Political-Military 
Affairs 


The  House  International  Relations 
Committee  (HIRC) 
Senate  Foreign  Relations  Committee 
(SFRC) 


E-IMET 
Program 


Under  Secretary  of  Defense 
for  Policy 

Assistant  Secretary  of  Defense  (International 
Securitv  Affairs) 


Defense  Security  Cooperation 
Agency  (DSCA) 

Joint  Staff 

Unified  Commands 

Military  Departments 
(MILDEPs) 

U.S.  Embassy  Security  Assistance 
Officer  (SAO) 

House  Appropriations  Committee  for  Foreign 
Operations  (HACFO) 


Senate  Appropriations  Committee  for  Foreign 
Operations  (SACFO) 

Figure  2.  Individual  E-IMET  Program  Stakeholders 

a.  Executive  Office  of  the  President  Stakeholders 

The  primary  stakeholder  organization  within  the  Executive  Office  of 
President  is  the  National  Security  Council  (NSC).  Chaired  by  the  President,  the 
function  of  the  council  is  to  advise  the  President  with  respect  to  the  integration  of 
domestic,  foreign,  and  military  policies  relating  to  national  security  (DISAM,  1994).  The 
NSC  publishes  the  National  Security  Strategy  of  the  United  States  that  outlines  the 
Administration's  core  national  security  objectives  and  strategic  priorities.  This  document 
contains  the  Administration's  broad  goals  and  objectives  relating  to  security  assistance  ( A 
National  Security  Strategy  for  a  New  Century,  1997).  It  is  the  NSC,  with  guidance  from 
the  President,  that  identifies  core  national  security  assistance  objectives. 
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b.  State  Department  Stakeholders 

The  Secretary  of  State  provides  continuous  supervision  and  general 
direction  for  security  assistance,  including  military  education  and  training  programs 
(DSCA  Website,  1 998).  The  Department  of  State  (DoS)  affects  program  determination 
and  integration.  It  is  the  DoS  that  determines  whether  there  will  be  an  E-IMET  program 
for  a  country  and,  if  so,  its  scope  (DISAM,  1994).  The  DoS  of  is  also  responsible  for 
"ensuring  that  the  programs  are  effectively  integrated  both  at  home  and  abroad,  and  that 
the  foreign  policy  of  the  United  States  is  best  served  thereby"  (DISAM,  1994,  p.  79).  To 
guide  these  decisions  the  Secretary  of  State  develops  international  affairs  strategic  goals 
that  are  the  foundation  of  the  nation's  foreign  policy  (FY  1999  Security  Assistance 
Budget  Request,  1998). 

The  Under  Secretary  for  International  Security  Affairs  is  the  principle 
advisor  to  the  Secretary  of  State  and  the  focal  point  for  security  assistance  matters  within 
the  Department  (DISAM,  1994).  The  Under  Secretary  chairs  the  Security  Assistance 
Program  Review  Working  Group  (SAPRWG)  which  is  the  DoS's  principle  advisor}' 
body  with  policy  planning  and  review  functions.  As  an  interagency  group  with  wide 
representation  throughout  the  Executive  Branch,  the  SAPRWG  acts  as  the  central  security 
assistance  policy-making  body  for  the  Secretary  of  State  (DISAM,  1994). 

The  Bureau  of  Political-Military  Affairs  also  plays  a  key  security 
assistance  role.  It's  principle  function  is  to  advise  the  Secretary  of  State  on  issues  and 
policy  problems  arising  from  areas  where  foreign  policy  and  defense  policy  impinge  on 
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one  another  (DISAM,  1994).    The  Bureau  also  plays  a  key  role  in  the  administration  of 
the  IMET  program. 

A  diagram  of  the  State  Department's  organizational  security  assistance 
stakeholders  is  provided  in  Figure  3. 


Office  of  the  Secretary  of  State 


Under  Secretary  for  International  Security  Affairs 


SAPRW 


Bureau  of  Political-Military  Affairs 


Figure  3.  State  Department  Security  Assistance  Stakeholders 

c.  Defense  Department  Stakeholders 

The  Secretary  of  Defense  is  responsible  for  implementing  programs  to 
transfer  security  assistance  services.  The  Defense  Secretary  is  responsible  for  the 
performance  of  all  defense  functions  that  furnish  military  assistance,  education,  and 
training  (DISAM,  1994).  Department  of  Defense  (DoD)  strategic  guidance  is  published 
in  the  National  Military  Strategy  and  the  Quadrennial  Defense  Review  (QDR). 

Authority  and  control  over  all  DoD  security  assistance  matters  is  delegated 
to  the  Under  Secretary  of  Defense  for  Policy.  The  Assistant  Secretary  of  Defense 
(International  Security  Affairs)  (ASD/ISA).  however,  supervises  security  assistance 
programs  with  all  foreign  governments.    Through  the  Defense  Security  Cooperation 


■r 


Agency  (DSCA),  the  ADS/ISA  "interprets  executive  policy  and  develops  DoD  security 
assistance  policies  and  programs"  (DISAM,  1994,  p.  84). 

The  DSCA  is  established  as  a  separate  agency  of  the  DoD  under  the 
direction  and  control  of  the  Under  Secretary  of  Defense  for  Policy  and  receives  policy 
direction  and  supervision  from  ASD/ISA.  DSCA  handles  many  critical  E-IMET 
functions  including  (DISAM,  1994); 

•  Administration  and  supervision  of  E-IMET  planning  and  training; 


• 


Coordination  of  the  formulation  and  execution  of  the  E-IMET  program 
with  other  government  agencies; 

•  Developing  and  promulgating  E-IMET  procedures. 

Additional  stakeholders  exist  within  DoD.  The  Joint  Staff  organization 
determines  the  impact  of  security  assistance  programs  on  U.S.  programs  defense 
readiness,  making  it  a  key  participant  in  the  security  assistance  program  development  and 
review  process  (DISAM,  1994).  The  five  Unified  Commands  are  responsible  for  the 
conduct  of  the  U.S.  security  assistance  program  within  their  respective  regions  (DISAM, 
1994).  They  provide  critical  program  evaluation  "of  the  efficiency  and  effectiveness  of 
DoD  overseas  security  assistance  programs"  (DISAM,  1994,  p.  87).  Each  Military 
Department  (MILDEP)  maintains  a  separate  department  responsible  for  the  actual 
delivery  of  security  assistance  services  and  provides  recommendations  to  the  Defense 
Secretary  on  the  successful  conduct  of  security  assistance  programs. 

A  diagram  of  the  Defense  Department's  organizational  security  assistance 
stakeholders  is  provided  in  Figure  4. 
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Office  of  the  Secretary  of  Defense 


Under  SECDEF  for  Policy 


1- 


Asst.  SECDEF  (International  Security 
Affairs)  (ASD/ISA) 


1- 


MILDEPs 


Defense  Security  Cooperation  Agency  (DSCA) 
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Commands 


Figure  4.  Defense  Department  Security  Assistance  Stakeholders 

d.  Legislative  Branch  Stakeholders 

Congress  establishes  the  laws,  authorizes  the  programs,  appropriates  the 
funds,  and  oversees  the  role  that  security  assistance  plays  in  international  affairs. 
Specifically,  the  Congress  has  the  power  to  (DISAM,  1994): 

•  Establish  or  amend  basic  security  assistance  authorization  acts; 

•  Enact  security  assistance  appropriation  acts; 

•  Pass  Joint  Resolutions  to  permit  the  incurrence  of  obligations  to  earn'  on 
essential  security  assistance  program  activities  until  appropriation  action  is 
complete; 

•  Conduct  hearings  and  investigations  into  special  areas  of  interest,  to 
include  instructions  to  the  General  Accounting  Office.  Congressional 
Budget  Office,  and  Congressional  Research  Office  to  accomplish  special 
reviews; 

•  Ratify  treaties  that  may  have  security  assistance  implications. 
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The  basic  congressional  stakeholder  organization  is  the  committee. 
Traditionally,  powerful  Authorization  and  Appropriation  Committees,  as  well  as  their 
respective  Subcommittees,  are  the  dominant  stakeholders.  These  committees  provide 
program  oversight  during  annual  hearings  on  fiscal  year  foreign  operations  spending. 
The  key  Authorization  Committees  include  (DISAM,  1994): 

•  House  International  Relations  Committee  (HIRC) 

•  Senate  Foreign  Relations  Committee  (SFRC) 

The  key  Appropriation  Subcommittees  include  (DISAM,  1994): 

•  House  Appropriations  Subcommittee  for  Foreign  Operations  (HACFO) 

•  Senate  Appropriations  Subcommittee  on  Foreign  Operations  (SACFO) 

2.  Stakeholder  Issues,  Concerns,  or  Stakes 

Once  the  key  stakeholders  have  been  identified  it  is  necessary  to  determine  the 
"stake"  each  has  in  E-IMET  effectiveness  evaluation.  According  to  Roberts  and  King, 

A  stake  is  the  claim  on  the  organization  that  each  stakeholder  has.  The 
concept  of  a  stake  is  based  on  the  idea  of  one's  having  something  to  lose 
or  gain  in  a  given  situation,  and  therefore  the  nature  of  the  stake  depends 
on  the  issue  at  hand.  Thus,  the  stake  may  be  tangible  (money,  material, 
resources)  or  intangible  (time,  prestige,  self-esteem),  explicit  or  implicit. 
Moreover,  a  stake  may  be  economic,  political,  social,  or  psychological  in 
nature  (Roberts  and  King,  1989,  p.  66). 

a.  The  Grouping  Process 

To  simplify  the  analysis  of  stakes  the  individual  stakeholders  will 
immediately  be  placed  into  groups.  Since  the  individual  stakeholders  are  organic  to 
higher  level  governmental  agencies  (i.e.,:     DSCA  is  organic  to  the  Department  of 
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Defense)  they  can  be  grouped  according  to  their  formal  organizational  charts.  It  is 
assumed  that  since  individual  stakeholders  are  under  operational  and  administrative 
control  of  their  higher-level  agency,  they  will  possess  a  similar  stake  in  the  evaluation  of 
the  E-IMET  program.  It  will  then  be  possible  to  characterize  the  stakes  of  each  group  and 
construct  an  overall  stakeholder  map.  The  following  stakeholder  groups,  along  with  their 
key  individual  stakeholders,  have  been  identified  for  this  study: 

The  Executive  Office  of  the  President  Group 

The  President 
The  National  Security  Council 

The  State  Department  Group 

The  Secretary  of  State 

The  Under  Secretary  for  International  Security  Affairs 

The  Security  Assistance  Program  Review  Working  Group 

The  Bureau  of  Political-Military  Affairs 

The  Defense  Department  Group 

The  Secretary  of  Defense 

The  Under  Secretary  of  Defense  for  Policy 

The  Assistant  Secretary  of  Defense  (ISA) 

The  Defense  Security  Cooperation  Agency 

The  Joint  Staff 

The  Unified  Commands 

The  Military  Departments 

The  Legislative  Group 

The  House  International  Relations  Committee 

The  Senate  Foreign  Relations  Committee 

The  House  Appropriations  Subcommittee  for  Foreign  Operations 

The  Senate  Appropriations  Subcommittee  for  Foreign  Operations 
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b.         Stakes  Analysis 

The  following  is  a  description  of  the  stakes  for  each  stakeholder  group.  It 
focuses  on  where  each  of  the  groups  stands  on  the  issue  of  E-IMET  performance 
evaluation. 

(1)  The  Executive  Office  of  President  Group.  The  President 
outlines  his  national  security  priorities  every  year  in  the  State  of  the  Union  Address.  In 
1997  President  Clinton  cited  the  following  security  assistance  priorities  (National 
Security  Strategy,  1997): 

•  Foster  a  peaceful,  undivided  Europe; 

•  Create  a  stable,  prosperous  Asia  Pacific  community'; 

•  Be  a  force  for  peace; 


• 


Maintain  a  strong  military  and  superb  diplomatic  representation  to  meet  all 
national  security  challenges. 

It  is  the  responsibility  of  the  National  Security  Council  to  develop 
the  national  security  strategy,  implementing  the  priorities  of  the  nation's  Chief  Executive. 
By  setting  national  goals  and  objectives,  the  Council  gives  direction  to  DoS  and  DoD 
security  assistance  efforts.  Thus,  the  Council  has  an  implicit  political  stake  in  the 
effectiveness  of  security  assistance  programs  like  E-IMET.  The  degree  to  which  a 
program  like  E-IMET  helps  accomplish  presidential  political  priorities  is  of  significant 
interest  to  the  President's  Executive  Office.  The  NSC  has  a  stake  in  measuring  the  E- 
IMET  program's  ability  to  accomplish  the  President's  strategic  priorities. 
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(2)  The  State  Department  Group.  The  State  Depart-ment  is 
responsible  for  overseeing  the  development  and  integration  of  various  security  assistance 
programs  throughout  the  Executive  Branch.  The  Department  submits  the  annual  Security 
Assistance  Budget  Request  that  supports  each  of  the  U.S.  foreign  operations  programs.  It 
also  publishes  the  annual  Congressional  Presentation  for  Foreign  Operations  (CPFO) 
that  supports  the  funding  requests  for  international  security  assistance  programs, 
including  the  IMET  program.  Since  current  E-IMET  expenditures  represent  only  a  small 
portion  of  overall  IMET  funding,  the  State  Department  is  concerned  primarily  with  the 
broader  IMET  program  (Bowab  Interview,  1998).  In  1998,  the  State  Department  claimed 
that  the  IMET  program  is  an  effective,  low-cost  component  of  the  global  U.S.  security 
assistance  effort  (CPFO,  1999): 

The  International  Military  Education  and  Training  (IMET)  program  is  an 
instrument  of  U.S.  national  security  and  foreign  policy — a  key  component 
of  U.S.  security  assistance.  It  is  a  program  that,  for  a  relatively  modest 
investment,  presents  democratic  alternatives  to  key  foreign  military  and 
civilian  leaders  (CPFO,  1999). 

E-IMET  funding,  however,  has  been  growing  steadily  as  a 
percentage  of  IMET  spending  since  the  1991  expansion  of  the  IMET  program.8  The 
State  Department,  therefore,  has  a  tangible  stake  in  assessing  the  effectiveness  of  the  E- 
IMET  program.  A  favorable  E-IMET  performance  evaluation  would  demonstrate  to 
Congress  the  cost  effectiveness  of  the  overall  IMET  program.    A  rigorous  performance 


8    In  1998,  Congress  specified  that  approximately  30%  of  total  IMET  funding  should  be  dedicated  to  the 
E-IMET  program  (Bowab,  1998). 


evaluation  that  yields  unfavorable  results,  however,  would  be  problematic  in  terms  of 
future  congressional  support. 

A  favorable  E-IMET  performance  evaluation  also  presents  the 
State  Department  with  a  potential  problem.  In  today's  environment  of  fiscal  constraint  a 
positive  performance  evaluation  could  also  be  used  to  justify  the  expansion  of  the  E- 
IMET  program  at  the  expense  of  traditional  IMET  training.  For  those  who  believe  that 
the  current  focus  on  E-IMET  training  is  taking  away  from  the  State  Department's  more 
traditional  military-to-military  IMET  training,  a  positive  review  of  the  E-IMET  program 
could  cause  an  unwanted  shift  away  from  the  IMET  program. 

(3)  The  Defense  Department  Group.  The  Defense  Depart- 
ment is  responsible  for  the  implementation  of  congressionally  funded  military  security 
assistance  programs.  Specifically,  the  DSCA  is  tasked  to  provide  the  services  associated 
with  the  E-IMET  program.  The  Secretary  of  Defense's  Report  of  the  Quadrennial 
Defense  Review  (QDR)  and  the  Joint  Staffs  National  Military  Strategy  outline  a  security 
strategy  of  engagement  for  the  Defense  Department  and  provide  a  mission  for  DSCA 
(QDR,  1997).  DoD's  strategy  of  engagement  emphasizes  the  role  of  security  assistance 
in  shaping  the  changing  global  security  environment. 

A  strategy  of  engagement  presumes  the  United  States  will  continue  to 
exercise  strong  leadership  in  the  international  community,  using  all 
dimensions  of  its  influence  to  shape  the  international  security 
environment.  This  is  particularly  important  to  ensuring  peace  and  stability 
in  regions  where  the  United  States  has  vital  or  important  interests  and  to 
broadening  the  community  of  free-market  democracies.  Strengthening 
and  adapting  alliances  and  coalitions  that  serve  to  protect  shared  interests 
and  values  are  the  most  effective  ways  to  accomplish  these  ends  (QDR, 
1997). 
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Through  peacetime  engagement  activities,  U.S.  Armed  Forces  promote 
regional  stability,  increase  the  security  of  allies  and  friends,  build 
coalitions,  and  ensure  a  more  secure  global  environment.  The 
International  Military  Education  and  Training  Program  (IMET) 
establishes]  long-term  professional  relationships  between  our  Armed 
Forces  and  the  future  military  leadership  of  other  countries  (National 
Military  Strategy,  1998). 

Similar  to  the  State  Department,  the  Defense  Department  has  an 
explicit  tangible  stake  in  measuring  the  performance  of  DoD  security  assistance 
programs.  Like  the  DoS,  DoD  is  subject  to  GPRA.  The  requirement  to  measure  the 
performance  of  security  assistance  programs  like  E-IMET  presents  DoD  with  a  double- 
edged  sword.  A  favorable  measurement  of  effectiveness  might  guarantee  future  funding. 
but  a  negative  report  could  jeopardize  the  program. 

The  Department  of  Defense  also  has  an  intangible  stake  in 
measuring  E-IMET  program  effectiveness.  As  the  international  security  environment 
develops  DoD  struggles  to  clearly  identify  its  changing  mission.  With  its  growing  role  in 
non-traditional  military  functions,  such  as  humanitarian  assistance  and  peacekeeping 
operations,  a  key  future  DoD  mission  might  be  the  delivery  of  international  military-to- 
military  and  military-to-civilian  training.  One  might  conclude  that  DoD's  future 
relevance  partially  rests  on  ensuring  the  continued  support  for  military  security  assistance 
programs  like  E-IMET.  A  favorable  E-IMET  performance  evaluation  would  go  a  long 
way  towards  assuring  continued  congressional  support  for  DoD  security  assistance 
programs. 
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(4)  The  Legislative  Group.  In  1993  President  Clinton  signed 
into  law  the  Government  Performance  and  Results  Act.  Also  in  1993,  the  President  and 
Vice  President  initiated  the  National  Performance  Review  (NPR)  to  reinvent  government 
(NPR  Benchmarking  Report,  1997).  As  Hayward  points  out,  however,  "GPRA  is  more 
than  an  administration  directive  that  can  be  rescinded  or  whose  implementation  can  be 
stalled  while  waiting  for  a  new  administration  with  a  different  agenda"  (Hayward,  1998, 
p.  2).  In  fact,  Congress  aspires  to  completely  enhance  government  performance  and 
results  (Hayward,  1998).  Representative  Deborah  Pryce  of  Ohio  states  very  clearly  the 
stake  the  Legislative  Branch  has  in  government  performance  evaluation: 

As  the  custodians  of  our  nation's  purse  strings,  Congress  has  an  incredible 
responsibility.  We  have  been  entrusted  to  safeguard  the  hard-earned 
money  that  the  taxpayers  send  to  Washington.  It  is  our  responsibility  to 
see  to  it  that  those  dollars  are  spent  wisely  and  that  the  American  people 
get  the  biggest  bang  for  this  buck  (freedom.house.gov/results  Website, 
1998). 

Many  other  Congressional  Senators  and  Representatives  share 
similar  feelings  (see  Armey,  Burton,  Horn,  and  Kingston,  freedom.house.gov/results 
Website,  1998).  In  fact,  the  House  Majority  Leader  has  an  Internet  Website  supporting 
and  tracking  the  progress  of  the  Results  Act  (http://freedom.house.gov/results). 
Improving  the  public  confidence  in  the  federal  government  by  systematically  holding 
federal  agencies  accountable  for  achieving  program  results  has  received  broad  support 
within  the  Legislative  Branch. 

Congressional  support  for  GPRA  represents  an  explicit  intangible 
stake  in  E-IMET  performance  evaluation.    Congress  has  clearly  voiced  their  support  for 
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legislation  that  holds  agencies  accountable  for  results.  Failure  to  follow  through  on  their 
promise  would  be  a  serious  blow  to  the  esteem  and  prestige  of  the  people's  elected 
officials.  Regularly  elected  government  officials  have  an  incentive  to  see  that  the 
constituency  they  represent  receives  value  for  their  tax  dollars. 

(5)  Summary'.  Four  different  groups  of  stakeholders  have  a 
stake  in  E-IMET  performance  evaluation.  The  Executive  Office  of  the  President  has  a 
political  stake  in  determining  whether  the  E-IMET  program  meets  the  President's 
national  security  priorities.  The  Departments  of  State  and  Defense  are  each  committed  to 
the  role  of  security  assistance  in  the  changing  global  security  environment.  Each 
department  is  also  subject  to  the  1993  Government  Performance  and  Results  Act.  Both 
departments  have  a  stake  in  demonstrating  the  effectiveness  E-IMET  in  order  to  ensure 
the  program's  continued  existence.  While  the  State  Department  tends  to  focus  on  the 
effectiveness  of  the  broader  IMET  program,  the  Defense  Department  recognizes  E-IMET 
as  a  separate  program  requiring  its  own  set  of  performance  indicators.  The  Legislature 
has  a  stake  in  the  government-wide  adherence  to  the  1993  Government  Performance  and 
Results  Act.  The  people's  elected  representatives  want  to  prove  to  their  constituents  that 
their  money  is  not  being  wasted  in  Washington. 

C.         THE  STAKEHOLDER  MAP 

Based  on  their  different  stakes  in  the  evaluation  of  the  E-IMET  program  the 
various  stakeholder  groups  can  finally  be  organized  into  an  overall  stakeholder  map. 
Figure  5  contains  the  E-IMET  stakeholder  map. 
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The  Executive  Office  of  the 
President  Group: 

The  President 
-      NSC 


The  Legislative  Group 


House  International 
Relations  Committee 
(HIRC) 

Senate  Foreign 
Relations  Committee 
(SFRC) 

House  Appropriations 
Subcommittee  for 
Foreign  Operations 
(HACFO) 

Senate  Appropriations 
Subcommittee  on 
Foreign  Operations 
(SACFO) 


E-IMET 
Program 


The  Defense  Department 
Group 

-  SecDef 

Under  SecDef  for  Policy 
Asst.  SecDef  (International 
Security  Affairs) 

-  DSCA 
Joint  Staff 
Unified  Commands 

-  MILDEPs 

U.S.  Embassy  Security 
Assistance  Officer 


The  State  Department  Group: 

SecState 

Under  SecState  for 
International  Security 
Affairs 
-      SAPRWG 

Bureau  of  Political- 
Military  Affairs 


KEY: 

Executive  Branch  Stakeholder  Groups 
Legislative  Branch  Stakeholder  Group 


Figure  5.  The  E-IMET  Stakeholder  Map 


D.         CHAPTER  SUMMARY 

The  development  and  management  of  the  E-IMET  program  involves  the  active 
participation  and  cooperation  of  several  key  stakeholders  within  the  national  government. 
The  National  Security  Council  of  the  President's  Executive  Office  establishes  the 
National  Security  Strategy  that  guides  all  other  Executive  Branch  security  assistance 
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efforts.  By  law,  the  Secretary  of  State  is  responsible  for  the  supervision  and  general 
direction  of  all  U.S.  security  assistance  efforts.  The  Department  of  Defense  is  responsible 
for  implementing  military  programs  such  as  E-IMET.  Through  the  DSCA.  the  DoD  *'has 
the  largest  supportive  role  from  a  level-of-effort  standpoint"  and  is  the  most  critical  E- 
IMET  stakeholder  (DISAM,  1994,  p.  95). 

E-IMET  stakeholders  also  exist  in  the  Legislative  Branch  of  the  national 
government.  Through  its  oversight  role  Congress  authorizes  and  appropriates  public 
funds  necessary  to  keep  the  program  alive.  In  an  effort  to  ensure  responsible  spending 
Congress  has  demanded  performance-based  budgeting  on  the  part  of  all  government 
agencies. 

Because  it  is  assumed  that  individual  stakeholders  within  the  same  higher-level 
governmental  agency  share  common  stakes  in  E-IMET  performance  evaluation,  a 
grouping  of  stakeholders  according  to  similar  stakes  is  possible.  Figure  5  represents  the 
stakeholder  map  for  the  E-IMET  program.  Figure  6  summarizes  the  flow  of 
communications  for  security  assistance  issues  from  the  perspective  of  the  Defense 
Security  Cooperation  Agency  (DSCA  Presentation.  1998). 


59 


The  President 


Secretary  of  State 


Secretary  of  Defense 
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Assistance  Officer 

In-Country 


Figure  6.  Security  Assistance  Communications  (DoD  Perspective) 
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V.         ANALYSIS:  PARTI 

A.         INTRODUCTION 

The  Government  Performance  and  Results  Act  of  1993  states  that  "federal 
managers  are  seriously  disadvantaged  in  their  efforts  to  improve  program  efficiency  and 
effectiveness,  because  of  insufficient  articulation  of  program  goals  and  inadequate 
information  on  program  performance"  (GPRA,  1993,  p.  2).  GPRA  emphasizes  the 
setting  of  clear  program  goals  and  the  measurement  of  program  performance  against 
those  goals  (GPRA.  1993).  Developing  a  clear  set  of  agreed-upon  program  goals 
(objectives)  is  often  problematic,  however.  As  Chapter  IV  clearly  demonstrates,  the  E- 
IMET  program  has  many  internal  and  external  stakeholders.  It  is  possible  that  each 
stakeholder  defines  the  program's  objectives  differently.  If  a  common  definition  of 
program  objectives  does  not  exist,  than  any  attempt  to  objectively  measure  program 
performance  will  ultimately  fail  as  multiple  constituencies  quarrel  over  whose  set  of 
objectives  will  serve  as  the  performance  baseline.  Thus,  it  is  necessary  to  analyze 
individual  stakeholder  objectives  in  order  to  determine  if  a  single  set  of  program 
objectives  exist. 

The  purpose  of  this  chapter  is  to  identify  each  stakeholder's  security  assistance 
objectives  to  determine  if  a  single  set  of  common  objectives  emerge.  Using  written 
documentation  from  each  group  of  stakeholders,  security  assistance  objectives  were 
singled-out  and  compared  to  other  stakeholder  groups.  Ultimately,  a  single  set  of 
national    security   assistance   objectives   emerged   and   was   arranged    into    a   "Global 


Hierarchy."  Specific  attributes  were  then  identified  for  each  of  the  common  global  sub- 
objectives  and  used  to  link  the  E-IMET  program  objectives  back  to  the  Global  Hierarchy, 
creating  an  objective  baseline  for  future  performance  measurement  efforts. 

B.         THE  GLOBAL  HIERARCHY 

1.  Methodology 

The  bulk  of  this  chapter  will  present  the  logic  used  to  construct  the  Global 

Hierarchy.  It  will  outline  the  steps  in  the  process,  highlight  the  key  assumptions  that 
were  necessary,  and  examine  the  difficulties  encountered  during  the  process.  The  chapter 
will  culminate  with  a  Global  Objectives  Hierarchy  for  security  assistance. 

a.         Step  #1:  Document  Review 

In  order  to  develop  a  Global  Objectives  Hierarchy  for  security  assistance 
one  must  start  with  a  detailed  review  of  the  key  stakeholder  documents  that  contain  the 
stakeholder's  security  assistance  objectives.  A  basic  assumption  is  that,  since  the  role  of 
the  Executive  Branch  of  the  national  government  is  to  set  policy,  only  Executive  Branch 
stakeholder  documents  should  be  used  to  form  the  Global  Hierarchy.  The  following 
Executive  Branch  stakeholder  documents  were  analyzed: 

Executive  Office  of  the  President: 

1 997  National  Security  Strategy 
1997  State  of  the  Union  Address 

Department  of  State: 

Fiscal  Year  1999  Security  Assistance  Budget  Request 

1998  Department  of  State  Strategic  Plan  for  International  Affairs 

1 998  Congressional  Presentation  for  Foreign  Operations 

Fiscal  Year  1 999  Department  of  State  Performance  Plan 
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Defense  Department: 

Fiscal  Year  1999  Support  for  the  Security  Assistance  Budget  Request 

1 997  Quadrennial  Defense  Review 

National  Military  Strategy 

Fiscal  Year  1999  Department  of  Defense  Performance  Plan 


The  preceding  list  of  documents  is  not  exhaustive.  It  is,  however,  a  fair 
representation  of  the  key  documents  that  guide  Executive  Branch  stakeholder  security 
assistance  policy.  In  addition,  as  with  the  document  selection  process,  all  remaining  steps 
used  to  construct  the  Global  Hierarchy  concentrates  only  on  Executive  Branch 
stakeholders. 

b.  Step  #2:  The  Sifting  Process 

Once  the  key  stakeholder  documents  were  chosen  a  sifting  process 
separating  security  assistance  objectives  from  other  unrelated  objectives  followed  for 
each  stakeholder  group.  This  process  is  largely  subjective,  although  many  of  the 
documents  clearly  identified  objectives  as  being  security  assistance  related.  Great  care 
was  taken  to  include  all  specific  security  assistance  objectives  as  well  as  those  that  have 
only  a  marginal  relation  to  the  stakeholder's  security  assistance  role.  Ultimately,  a 
separate  objectives  hierarchy  was  constructed  for  each  stakeholder  group  using  the  sifting 
process  and  the  selected  documents. 

The  result  was  three  stakeholder  group  hierarchies  from  which  to  form  one 
Global  Hierarchy.  The  purpose  of  first  constructing  three  individual  hierarchies  was  to 
allow  for  a  comparison  of  security  assistance  objectives  among  the  stakeholder  groups. 
Objectives  consensus  is  the  foundation  for  the  Global  Hierarchy  and  the  key  prerequisite 
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for  objective  performance  evaluation  of  the  E-IMET  program.  By  comparing  the 
hierarchies  to  one  another  it  is  possible  to  determine  the  degree  of  consensus  among  the 
stakeholders'  objectives. 

(1)  Sifting  Process  Challenges.  Extracting  specific  security 
assistance  objectives  from  the  key  stakeholder  documents  was  a  difficult  and  highly 
subjective  process.  Four  specific  difficulties  were  encountered  during  the  sifting  process. 

(a)  Separating  Objectives  from  Alternatives:  Ends  versus 
Means.  A  common  problem  found  in  the  key  stakeholder  documents  was  the  mixing  of 
objectives  with  specific  strategies  available  to  accomplish  those  objectives.  Few  federal 
documents  systematically  break  down  broad  objectives  into  their  relevant  sub-objectives. 
Instead,  many  of  the  stakeholder  documents  highlighted  a  broad  objective  ("ends")  and 
proceeded  directly  into  a  discussion  of  the  alternatives  available  to  achieve  the  final  result 
desired  ("means").  An  example  can  be  found  in  the  Defense  Department's  FY99 
Performance  Plan.  Promote  Stability,  which  is  an  ends  or  objective,  is  immediately 
followed  by  Overseas  Presence,  which  is  a  means  or  method  for  accomplishing  the 
objective  (DoD  Performance  Plan,  1998).  This  confusion  inevitably  results  in  a  lack  of 
measurable  detail  concerning  the  overall  objective  that  is  required  if  one  is  to  determine 
whether  the  program  achieved  its  objectives.  In  other  words,  while  government  agencies 
are  busy  identifying  methods  to  achieve  an  objective,  they  fail  to  sufficiently  expand 
enough  on  their  objectives  to  make  measurement  possible. 

(b)  Separating  Security  Assistance  Objectives  from 
Unrelated  Objectives.  Extracting  from  the  key  stakeholder  documents  the  objectives  and 

64 


sub-objectives  that  relate  only  to  security  assistance  was  the  most  difficult  part  of 
constructing  the  Global  Hierarchy.  Each  document  contained  references  to  stakeholder 
objectives  that  are  not  specifically  related  to  their  security  assistance  mission.  An 
example  can  be  found  in  the  Department  of  State's  FY99  Performance  Plan.  The  State 
Department  lists  Stabilize  World  Population  Growth  as  a  global  objective.  Clearly  this 
objective  is  not  related  to  the  definition  of  security  assistance  from  Chapter  I,  and  should 
not  be  included  in  a  global  security  assistance  hierarchy. 

Again,  this  is  a  highly  subjective  process  as  many 
governmental  objectives  are  interdependent.  It  is  essential,  however,  to  separate  specific 
security  assistance  goals  from  unrelated  goals  in  order  to  construct  a  single  hierarchy. 

(c)  Practice  in  Constructing  a  Global  Hierarchy.  As  Saaty 
points  out,  in  practice  there  is  no  set  procedure  for  generating  the  objectives  and  sub- 
objectives  included  in  a  hierarchy  (Saaty.  1980).  In  fact,  constructing  a  global  security 
assistance  hierarchy  proved  to  be  an  original  exercise.  No  individual  interviewed  from 
any  of  the  Executive  or  Legislative  Branch  stakeholder  agencies  knew  of  any  previous 
effort  to  link  national  security  assistance  objectives  using  a  Global  Hierarchy.  The  lack 
of  practice  in  creating  a  Global  Objectives  Hierarchy  resulted  in  a  significant  challenge  to 
the  overall  process. 

(d)  Time  Commitment.  One  explanation  for  the  lack  of 
practice  in  constructing  a  Global  Hierarchy  is  the  amount  of  time  it  takes  to  analyze  the 
key  documents  for  each  E-IMET  stakeholder.  Since  responsibility  for  various  parts  of 
the  E-IMET  program  is  shared  among  three  primary  agencies  within  the  Executive 
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Branch,  the  process  of  constructing  individual  hierarchies  and  synthesizing  them  into 
one  Global  Hierarchy  was  very  time  consuming. 

c.  Step  #3:  Constructing  the  Global  Hierarchy 

The  next  step  is  to  actually  construct  the  Global  Hierarchy.  The  existence 
of  objective  consensus  among  the  individual  stakeholders  is  vital  to  the  construction  of  an 
objectives  hierarchy.  Fortunately,  once  ends  were  separated  from  their  means,  there 
emerged  a  consensus  of  basic  security  assistance  sub-objectives  among  Executive  Branch 
stakeholders. 

Each  stakeholder  defined  their  individual  objectives  in  relation  to  The 
President's  three  core  national  security  objectives  of  enhanced  security,  economic 
prosperity,  and  a  democratic  world  (National  Security  Strategy  and  State  of  the  Union 
Address,  1997)  (See  Figure  7).  Further,  agreement  existed  among  the  remaining 
stakeholders  concerning  their  definition  of  the  sub-objectives  of  the  President's  core 
goals.  These  common  sub-objectives  include  international  stability,  uninhibited 
access  to  key  markets,  energy  supplies,  and  strategic  resources,  and  support  for 
democratic  principles  (See  Figure  7). 

The  next  step  in  the  construction  process  was  to  further  break  down  each 
broad  sub-objective  into  the  specific  attributes  that  characterize  that  sub-objective.  Here 
the  stakeholders  defined  the  specific  attributes  from  their  point  of  view.  For  instance,  the 
State  Department  recognized  that  uninhibited  access  to  key  markets,  energy  supplies,  and 
strategic  resources  require  the  growth  of  transitional  and  developing  economies  from 
socialist  to  market-based  economies.  The  Defense  Department,  however,  recognized  that 
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the  same  sub-objective  requires  access  to  foreign  military  bases.  Are  one  right  and  the 
other  wrong?  Within  the  context  of  the  Global  Hierarchy,  the  mission  of  national 
security  assistance  should  include  both  stakeholder  attributes.  Building  the  Global 
Hierarchy,  therefore,  is  a  process  of  incorporating  each  stakeholder's  relevant  attributes 
into  the  larger  picture.  This  is  exactly  as  GPRA  1993  dictates — the  inclusion  of  internal 
and  external  stakeholders  into  the  objectives-setting  process.  The  Global  Hierarchy, 
therefore,  starts  with  a  common  set  of  broad  sub-objectives  and  incorporates  each 
stakeholder's  relevant  attributes. 

The  final  Global  Objectives  Hierarchy  can  be  found  in  Figure  7. 

d.  Step  #4:  Validation 

Ultimately,  the  resulting  Global  Hierarchy,  and  the  criteria  used  to  select 
the  sub-objectives  and  specific  attributes,  was  validated  through  an  interview  process 
with  various  Executive  Branch  stakeholders.  The  validation  process  consisted  of 
interviews  with  officials  from  the  Departments  of  State  and  Defense.9  Officials  were 
chosen  from  the  organizations  within  each  stakeholder  group  that  deal  most  often  with  E- 
IMET  issues.  They  are  assumed  to  represent  the  views  of  the  entire  stakeholder  group. 
Interviews  were  conducted  with  officials  from  the  following  organizations: 

Department  of  State: 

The  Bureau  of  Political-Military  Affairs 


9  Officials  from  the  Executive  Office  of  the  President  were  unavailable  for  interviews.  The 
National  Security  Strategy  and  the  President's  State  of  the  Union  Address,  which  define  the  ultimate 
objectives  placed  on  top  of  the  Global  Hierarchy,  clearly  convey  the  views  of  that  key  stakeholder. 
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Figure  7 
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Specific  Attributes 


E-IMET  Objective  #1:  Respect  for  Civilian  Control  Over  the 
Military 


E-IMET  Objective  #2:  Improved  Military  Justice  Systems 
and  Procedures  In  Accordance  With 
International  Human  Rights 


E-IMET  Objective  #3:  Improved  Defense  Resources  Management 


Department  of  Defense: 

Asst.  Secretary  of  Defense  (International  Security  Affairs) 
Defense  Security  Cooperation  Agency 

( 1 )  Validating    the    Objectives    Selection    Criteria.       Two 

simple  criteria  were  used  to  select  the  individual  stakeholder  objectives  and  specific 
attributes  that  would  be  incorporated  into  the  Global  Hierarchy. 

The  first  criterion  was  that  the  stated  objective  had  to  fit  the 
definition  of  an  ""end."  In  other  words,  to  qualify  as  an  objective  it  could  not  be 
interpreted  as  "means,"  or  an  alternative.  It  must  be  a  clear  statement  of  the  final  result 
desired,  not  a  method  to  get  there. 

The  second  criterion  was  that  the  objective  had  to  relate  to  security 
assistance,  either  directly  or  indirectly.  This  criterion  is  purposefully  subjective  and 
broad  in  recognition  of  the  fact  that  security  assistance  takes  many  forms  and  is  related  to 
many  objectives. 

Each  stakeholder  interviewed  validated  the  criteria  used  to  select 
objectives  for  the  Global  Hierarchy. 

(2)  Validating  the  Global  Hierarchy.  Each  stakeholder 
group  interviewed  also  validated  the  final  Global  Hierarchy.  No  disagreement  existed 
among  the  stakeholders  as  to  the  relationship  between  the  highest  level  objectives  defined 
by  the  President,  the  common  sub-objectives,  and  the  specific  attributes  that  characterize 
the  sub-objectives. 
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It  should  be  stressed  that,  while  the  Global  Hierarchy  was  validated 
through  numerous  stakeholder  interviews,  national  security  assistance  objectives  will 
likely  change  over  time.  As  such,  the  Global  Hierarchy  will  require  periodic  revalidation 
as  international  security  conditions  evolve.  Revalidation  efforts  should  also  include  new 
stakeholders  whenever  possible. 

2.  Linking  the  E-IMET  Objectives  to  the  Global  Hierarchy 

Once  the  Global  Hierarchy  was  constructed  and  specific  attributes  were  identified 
the  next  logical  step  was  to  link  the  specific  E-IMET  program  objectives  back  to  the 
hierarchy.  If  an  E-IMET  objective  could  be  logically  linked  to  the  achievement  of  a 
specific  attribute  within  the  Global  Hierarchy  a  solid  triangle,  circle,  or  box  was  placed 
under  the  attribute.  This  linkage  is  made  possible  by  the  development  of  individual 
hierarchies  for  each  E-IMET  objective.  For  example,  improved  military  justice  systems 
in  accordance  with  international  human  rights  can  be  linked  to  the  existence  of 
neutral  law  enforcement  within  a  country  since  neutral  law  enforcement  bodies  are 
central  to  Western-style  military  justice  systems  (See  Figure  7). 

Because  performance  measurement  systems  do  not  yet  exist  within  the  security 
assistance  community,  any  attempt  to  link  E-IMET  objectives  back  to  the  Global 
Objectives  Hierarchy  is  largely  subjective.  Quantitative  data  have  yet  to  be  collected  that 
objectively  conclude  that  E-IMET  programs  have  resulted  in  the  achievement  of  higher 
global  objectives.  The  development  of  individual  hierarchies  for  each  E-IMET  objective, 
however,  provides  a  formal  logic  for  the  linkage.  Additionally,  anecdotal  evidence,  in  the 
form  of  congressional  testimony  from  various  State  and  Defense  Department  officials,  as 
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well  as  stakeholder  validation,  is  available,  however.10  Furthermore,  interviews  with  the 
Executive  Branch  stakeholders  validated  the  logical  link  between  the  Global  Hierarchy 
and  specific  E-IMET  objectives. 

The  validated  Global  Hierarchy  and  its  connection  to  the  specific  E-IMET 
objectives  can  be  found  in  Figure  7. 

C.         CHAPTER  SUMMARY 

Figure  7  contains  the  Global  Objectives  Hierarchy  for  security  assistance.  To 
construct  the  hierarchy  a  sifting  process  was  applied  to  key  Executive  Branch  stakeholder 
documents  in  an  effort  to  identify  their  individual  security  assistance  sub-objectives  and 
specific  attributes.  While  the  President  defines  the  ultimate  security  assistance  objective, 
a  common  set  of  sub-objectives  was  identified  for  each  stakeholder  group.  To  complete 
the  hierarchy  each  stakeholder's  specific  attributes  for  the  common  sub-objectives  were 
added.  Thus,  in  the  spirit  of  the  Government  Performance  and  Results  Act  of  1993.  a 
Global  Hierarchy  emerged  that  incorporates  each  stakeholder  group  into  the  objectives- 
setting  process. 

Figure  7  also  shows  the  logical  link  between  specific  E-IMET  objectives  and  the 
national  security  assistance  objectives.  While  no  quantitative  performance  data  exists  to 
objectively  link  specific  E-IMET  objectives  back  to  the  Global  Hierarchy,  anecdotal 
evidence  is  available.  Now  that  specific  E-IMET  objectives  have  been  logically  linked  to 


10  Anecdotal  evidence  concludes  that  IMET  and  E-IMET  programs  are  an  integral  part  of  the  national 
security  assistance  effort.  Examples  include  congressional  testimony  from  military  officers,  including 
various  regional  Commanders-in-Chief  (CrNCs).  and  various  DoD  and  DoS  officials  (see  Larson  1993. 
McCaffrey  1995,  Wilhelm  1998,  Slocombe  1998,  Shattuck  1996,  and  Hamilton.  1997). 
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national  security  assistance  objectives  the  challenge  is  to  develop  performance  indices 
that  objectively  measure  the  effectiveness  of  the  E-IMET  intervention. 
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VI.        ANALYSIS:   PART  II 

A.  INTRODUCTION 

The  third  step  in  the  public  sector  performance  evaluation  process  is  the 
development  of  performance  indices  for  each  program  objective.  The  objectives  of  the  E- 
IMET  program  are: 

1 .  Respect  for  Civilian  Control  Over  the  Military; 

2.  Improve  Military  Justice  Systems  and  Procedures  In  Accordance  With 
International  Human  Rights; 

3.  Improve  Defense  Resources  Management. 

In  order  to  evaluate  whether  various  E-IMET  interventions  are  having  the  desired 
effect  one  must  measure  results  against  the  original  objectives.  This  chapter  will  identify 
specific  performance  indicators  that  can  be  used  to  evaluate  the  effectiveness  of  the  E- 
IMET  program  and  highlight  the  key  difficulties  encountered  in  the  process. 

The  indices  proposed  in  this  chapter  stem  from  the  data  collected  through 
secondary  archival  research  and  specialized  interviews.  It  is  important  to  note  that  the 
indices  developed  here  are  general  in  nature  and  have  not  been  patterned  to  a  specific 
country  or  region.  This  chapter  is  not  meant  to  be  an  exhaustive  list  of  E-IMET 
performance  indices,  but  rather  a  starting  point  for  future  development. 

B.  THE  IMPORTANCE  OF  A  BASELINE  ASSESSMENT 

The  performance  evaluation  process  is  essentially  a  comparison  of  post- 
intervention  results  against  objective  data  gathered  during  a  baseline  assessment.      \ 
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baseline  assessment  is  defined  as  the  "initial  collection  of  data  to  establish  a  basis  for 
comparison"  (National  Performance  Review,  1997,  p.  27).  The  success  of  the  E-IMET 
performance  evaluation  process  hinges  on  the  objective  initial  assessment  of  a  nation's 
civilian  leadership,  military  justice  system,  and  ability  to  efficiently  manage  defense 
resources.  If  one  does  not  possess  a  clear  picture  of  civil-military  relations,  military 
justice,  and  resources  management  within  a  country  before  the  initial  intervention, 
attributing  future  incremental  changes  to  the  E-IMET  program  is  not  possible.  All 
program  results  must  therefore  be  stated  in  terms  of  a  progression  away  from  the  baseline 
assessment. 

The  performance  indices  proposed  in  this  chapter  assume  that  an  objective 
baseline  assessment  is  conducted  by  U.S.  Embassy  staff  (i.e.:  The  Security  Assistance 
Officer  (SAO))  or  other  DoD  officials  before  a  country  is  selected  to  receive  E-IMET 
training.  Currently,  "needs  assessments"  are  routinely  conducted  as  part  of  the  first  phase 
of  many  E-IMET  programs.  It  is  unclear,  however,  whether  current  needs  assessments 
are  intended  to  form  the  objective  baseline  necessary  for  incremental  performance 
evaluation  or  simply  an  effort  to  match  specific  E-IMET  programs  to  perceived  needs 
within  a  country.  The  design  and  implementation  of  an  E-IMET  performance  evaluation 
baseline  assessment  is  beyond  the  scope  of  this  study,  but  its  importance  within  the 
performance  evaluation  process  should  not  be  overlooked. 

C.         INDICATORS  OF  PERFORMANCE  FOR  THE  E-IMET  PROGRAM 
1.  Civilian  Control  Over  the  Military 

The  first  E-IMET  objective  is  the  promotion  of  understanding  and  respect  for 
civilian  control  over  the  military.    In  order  to  propose  performance  indicators  for  this 
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objective  one  must  define  the  specifics  of  civilian  control  over  the  military. 
Unfortunately,  many  scholars  debate  the  characteristics  of  civilian  control  over  the 
military  in  terms  of  modern  civil-military  relations.1 1 

a.  Civilian  Control  versus  Good  Civil-Military  Relations 

The  congressional  legislation  authorizing  the  expansion  of  IMET  training 
to  include  civilian  control  over  the  military  concentrates  only  on  the  act  of  civilian 
control,  not  on  the  dynamics  of  good  civil-military  relations  (U.S.  Congress,  1991).  The 
difference  is  subtle,  but  enormously  important  in  terms  of  performance  evaluation. 
Evaluating  the  quality  of  a  country's  civil-military  relations  is  largely  subjective,  but 
evaluating  the  degree  of  civilian  control  over  the  military  is  fairly  clear  cut.  While 
scholars  disagree  on  what  conditions  are  best  for  good  civil-military  relations,  there  is 
rough  agreement  on  what  constitutes  civilian  control.  During  the  Cold  War  both  the 
United  States  and  the  Soviet  Union  enjoyed  civilian  dominance  over  their  expansive 
militaries,  but  scholars  often  characterize  U.S.  and  Soviet  civil-military  relations  quite 
differently  (see  Michael  C.  Desch's  "Threat  Environments  and  Military  Missions."  in 
Diamond  and  Plattner's  Civil-Militarv  Relations  and  Democracy,  1996).  E-IMET 
performance  indices,  therefore,  must  concentrate  only  on  the  changes  within  a  country 
that  results  in  greater  civilian  control  over  the  military  establishment.  Whether  gains 
toward  civilian  control  result  in  better  overall  civil-military  relations  and  a  more  peaceful 
or  democratic  society  is  another  question  entirely. 


1 '  The  writings  of  Huntington  1996,  Desch  1996  and  1998,  Avant  1998.  Bracken  1995,  and  Segal  1995 
offer  lively  debates  concerning  civilian  control  over  the  military  and  modern  civil-military  relations. 
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b.  Civilian  Control  Objectives  Hierarchy  and  Performance  Indices 

In  order  to  develop  performance  indices  for  civilian  control  over  the 
military,  one  must  first  break  down  the  broad  objective  of  civilian  control  into  multiple 
sub-objectives  and  specific  attributes.  The  goal  of  this  process  is  to  construct  a  civilian 
control  over  the  military  objectives  hierarchy  that  can  be  used  to  identify  subsequent 
performance  indicators.  Drawing  heavily  from  the  U.S.  model  of  civilian  military 
control,  and  following  the  same  format  as  the  Global  Security  Assistance  Hierarchy 
found  in  Chapter  V,  the  objective  of  civilian  control  over  the  military  can  be  broken  down 
into  the  following  sub-objectives: 

Sub-Objective  #1: 
Military's  Organizational  Structure  and  Culture  Emphasize  Civilian  Dominance 

Sub-Objective  #2: 
The  Military  is  used  to  Counter  External  Threats  vice  Internal  Problems 

Sub-Objective  #3: 
Presentation  of  the  Military  is  Through  an  Objective  Source12 

Specific  attributes  that  characterize  each  sub-objective  must  then  be 
identified  in  order  to  complete  the  hierarchy.  Based  on  the  specific  attribute,  it  is  possible 
to  propose  indicators  of  performance  that  can  be  used  to  report  whether  the  E-IMET 
intervention  is  achieving  its  stated  objective.  The  specific  attributes  of  each  sub- 
objective,  along  with  their  respective  indicators  of  performance,  follow. 


^2  This  sub-objective  refers  to  the  right  of  the  citizenry  to  receive  information  concerning  the  activities  and 
policies  of  their  military  without  state-sponsored  censure. 
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Sub-Objective  #/: 
Military's  Organizational  Structure  and  Culture  Emphasize  Civilian  Dominance 

Specific  Attributes: 

1.  The  Commander-in-Chief  of  the  armed  forces  is  a  civilian  and  shares 
power  with  a  popularly  elected  legislature. 

Indicators  of  Performance: 

•  The  Commander-in-Chief  commissions  officers  after  first  nomina- 
ting them  to  the  people's  popularly  elected  representatives. 

•  The  Commander-in-Chief  promotes  officers  after  first  nominating 
them  to  the  people's  popularly  elected  representatives. 

•  The  Commander-in-Chief  is  a  civilian. 

2.  Norms  exist  within  the  military  for  subordination  to  civilian  control. 

Note:      This  attribute  can  be   further  divided   into  the   following  two 
attributes: 

a.  The  military  is  politically  neutral. 

Indicators  of  Performance: 


• 


• 


• 


The  military  accepts  all  missions  laid  out  by  the  civilian 
authorities. 

The  military  does  not  try  to  influence  policy-making 
(specific  limitations  and  constraints  are  placed  on  the 
military's  ability  to  influence  policy  debates). 

Civilians  decide  on  ends  of  policy  while  the  military 
decides  on  the  means,  and  civilian  authorities  define  the 
line  between  ends  and  means. 

The  military  is  professionally  vice  ideologically  driven. 

The  military  does  not  impose  its  values  upon  civilian 
institutions  and  organizations. 
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•  Special    military    bodies   exercising   political   power   are 
eliminated. 

b.  A  dialogue  exists  between  civilian  and  military  experts. 

Indicators  of  Performance: 

•  Open  debate  exists  within  military  and  civilian  commun- 
ities. 

•  Civilian  and  military  staffs  work  together  to  solve  long 
term  problems  and  issues. 

•  The    number    of  major   conflicts   between    civilian   and 
military  leaders  decreases. 

•  The  public  has  access  to  military  leaders  through  their 
elected  representatives. 

The  civilian  leadership  maintains  total  legal  control  over  the  military. 

Note:     This  attribute  can  be  further  divided  into  the  following  two 
attributes: 

a.  The   military's   budget  is   controlled   by  the   popularly   elected 

legislature. 

Indicators  of  Performance: 

•  The    legislature    approves    the    military's    budget    and 
subsequent  expenditures  ("The  Power  of  the  Purse"). 

•  The   legislature   approves   the   final   composition   of  all 
military  bases  and  depots. 

•  The  overall  size  of  the  military  is  controlled  by  the  civilian 
legislature. 

•  The  legislature  approves  the  composition  of  forces. 
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b.  The  civilian-military  chain  of  command  is  legally  based. 

Indicator  of  Performance: 

•  A  formal  civilian-controlled  chain  of  command  exists  and 
is  based  in  law. 

Civil-military  organizational  relationships  emphasize  civilian  control. 

Note:      This  attribute  can  be  further  divided   into  the   following  two 
attributes: 

a.  The  civilianization  of  high  political  offices. 

Indicator  of  Performance: 

•  Movement  of  civilians  into  traditional  military  positions 
within  the  Defense  Ministry. 

b.  No  excessive  influence  by  the  military  of  the  military-industrial 
complex. 

Indicators  of  Performance: 


• 


The  legislature  controls  the  military's  relationship  with 
industry. 


•  Military-run  business  and  industries  are  privatized. 

Sub-Objective  #2: 
The  Military  is  used  to  Counter  External  Threats  vice  Internal  Problems 

Specific  Attributes: 
1 .  Externally  oriented  military  mission  and  doctrine. 

Indicators  of  Performance: 
•  Military  doctrines  are  implemented  with  an  external  focus. 
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•  The  curricula  of  service  academies  are  revised  stressing  civilian 
control  and  external  missions  for  the  military. 

•  Laws  regulating  the  use  of  the  military  for  internal  purposes  are 
implemented. 

•  The  military  redeploys  forces  away  from  internal  missions. 

•  The  military  defends  society  against  external  threats. 

2.  Internal  problems  are  addressed  without  the  use  of  the  military. 

Indicator  of  Performance: 

•  Internal  problems,  formerly  addressed  through  military  means,  are 
now  addressed  with  non-military  state  agencies. 

Sub-Objective  #3: 

Presentation  of  the  Military  is  Through  an  Objective  Source 

Specific  Attributes: 

1 .  A  free  domestic  media. 

Indicators  of  Performance: 

•  Open  dialogue  exists  between  the  press  and  the  military. 

•  Governmental     control     (censorship)     of    press     reporting     is 
discontinued. 

2.  Free  public  access  to  published  government  information. 

Indicators  of  Performance: 

•  The  public  has  free  access  to  military  budget  data  and  deployment 
statistics. 

•  The  public  has  free  access  to  the  voting  records  of  elected  officials. 
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The  complete  hierarchy  of  the  sub-objectives  and  specific  attributes  for  the 
civilian  control  objective  can  be  found  in  Figure  8. 


>> 

I 


.v.   o' 
>  '-3 

©is* 
go 

|;| 

s-a 

in    O 

ublic 
.s  to 
,hed 

!.| 

FreeP 

Acce 

PublL 

Goverru 

Inform 

u 

1.= 

•»:£■•§ 

Swl 
« .JiA 

.^.l  ■■■■• 

■■  /  o  ■  , 


2  .3  J  -§ 

u<5  a: 


■  ■§  -i  -  fc  - 
J5 .1  j.5  is 


V, 


c 

.9  -a 


«   5  .S 


«  •- 

'.S  r 


o|iiGl 


'  CO 


> 

u 

u 

> 

3 

> 

X.' 

G 

< 

w 

H 

S" 

o 

£ 

c 

G 

5 

^0 

u 

CO 

DO 

The 
Civilian- 
Military 
Chain-of- 
Command 
is  Legally 
Based 

9  «*  £ 


1      £*      S 


.a 


•S  -S  U.  rj        o   „,       3 


Figure  8.  Civilian  Control  Over  the  Military  Objectives  Hierarchy 


c.  Limitations 

While  the  researcher  believes  that  objectively  measuring  the  progress 
toward  civilian  control  over  the  military  is  possible,  limitations  do  exist.  First,  many  of 
the  performance  indicators  measure  only  structural  changes  in  the  military  organization. 
Cultural  and  behavioral  changes  are  often  more  important  and  much  more  difficult  to 
measure.  This  limitation  should  not  overshadow,  however,  the  importance  of  detecting 
structural  changes,  as  they  are  necessary  in  order  to  bring  about  the  changes  in  behavior 
that  characterize  the  culture  of  civilian  control. 

A  second  limitation  involves  the  long-term  nature  of  organizational 
change.  Segal  warns  that  that  civilian  control  and  good  civil-military  relations  is  a 
process,  not  an  existential  state  (Segal,  1995).  Attempts  to  measure  civilian  control  over 
the  military  should  not  be  single  snapshots  in  time  but  rather  a  series  of  snapshots  over 
the  duration  of  the  program.  Each  measure  must  be  continually  compared  to  the  baseline 
assessment  in  order  to  determine  if  the  E-IMET  intervention  had  the  intended  effect. 

Another  important  limitation  is  the  inability  to  control  external  variables 
when  measuring  the  effectiveness  of  the  E-IMET  intervention.  Huntington  points  out 
that  one  reason  why  so  many  military  regimes  in  Latin  America  have  voluntarily 
surrendered  power  is  due  not  to  the  efforts  of  the  U.S.,  but  rather  to  the  recognition  on  the 
part  of  military  leaders  "that  there  are  no  easy  solutions  to  the  intractable  economic, 
social,  and  political  problems  confronting  their  respective  countries,  and  that  the 
sustained  involvement  in  politics  has  disastrous  effects  on  the  coherence,  efficiency,  and 
discipline  of  the  army"  (Huntington,  1995,  p.  7).  Military  leaders  selfishly  chose  to  avoid 
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politics  in  an  effort  to  regain  control  over  their  armies  and  leave  the  tough  decisions  to 
others.  In  this  case,  survival,  rather  than  a  dedication  to  the  ideal  of  civilian  control. 
motivates  the  military's  surrender  of  power.  Situations  such  as  this  make  it  difficult  to 
separate  the  effects  of  the  E-IMET  intervention  from  the  external  environment. 

Contentious  indicators  of  civilian  control  over  the  military  also  exist  that 
make  performance  evaluation  more  complicated.  Scholars  disagree,  for  example, 
whether  military  representativeness  of  society  is  an  indicator  of  civilian  control. 
Huntington  believes  that  a  professional  and  politically  neutral  military  is  an  indicator  of 
civilian  control  and  proper  civil-military  relations,  regardless  of  its  societal  representation 
(Huntington,  1995).  Janowitz,  on  the  other  hand,  believes  that  proper  civilian  control 
starts  with  the  "citizen-soldier"  and  a  military  that  mirrors  the  society  it  defends  (Segal. 
1995).  For  this  reason,  the  researcher  chose  not  to  include  military  representativeness  of 
society  as  an  indicator  of  performance.  Other  contentious  indicators,  such  as  the 
abolition  of  conscription,  were  also  intentionally  left  out  because  it  is  unclear  that  they 
indicate  greater  civilian  control  over  the  military. 

2.  Improved  Military  Justice  Systems 

The  1991  law  that  expanded  IMET  training  specifically  called  for  new  courses 
focused  on  "creating  and  implementing  effective  military  justice  systems  and  codes  of 
conduct  with  a  special  emphasis  on  the  protection  of  human  rights"  (U.S.  Congress.  1991, 
p.  135).  The  new  law  called  for  a  dedicated  training  effort  aimed  at  improving  the 
military  justice  systems  of  current  and  potential  U.S.  allies  in  accordance  with 
international  human  rights  standards.  As  with  civilian  control  over  the  military,  assessing 


the  progress  toward  this  objective  is  a  matter  of  measuring  post-training  gains  toward 

more  humane  military  justice  systems. 

a.  Improved  Military  Justice  Systems  Objectives  Hierarchy  and 

Performance  Indices 

In  order  to  propose  performance  indicators  for  this  objective  one  must 
define  the  essential  characteristics  and  objectives  of  a  humane  system  of  military  justice. 
The  U.S.  system  of  military  justice  will  serve  as  the  model  for  this  study.  Using  the 
objectives  hierarchy  approach  the  broad  objective  of  improved  military  justice  systems 
will  be  broken  down  into  multiple  sub-objectives  and  specific  attributes  patterned  after 
the  U.S.  model  of  military  justice.  The  goal  of  this  process  is  to  construct  an  improved 
military  justice  systems  objectives  hierarchy  that  can  be  used  to  identify  subsequent 
performance  indicators.  The  improved  military  justice  systems  objective  can  be  broken 
down  into  the  following  broad  sub-objectives: 

Sub-Objective  #1: 
Eliminate  the  Potential  for  War  Crimes 

Sub-Objective  #2: 
Eliminate  Peacetime  Human  Rights  Abuses  by  the  Military 

The  second  sub-objective  can  be  further  broken  down  into  the  following 

two  additional  objectives: 

Additional  Objective  A: 

Establish  a  Western  Style  Military  Justice  System  That  Recognizes  the 
Rights  of  the  Individual 
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Additional  Objective  B: 
Civilian  Control  Over  the  Military 

In  this  case  sub-objective  #1  shares  common  attributes  with  both  the 
additional  objectives  identified  for  sub-objective  #2.13  The  following  common  specific 
attributes,  along  with  their  respective  indicators  of  performance,  have  been  identified  for 
the  additional  objectives  of  sub-objective  #2: 

Additional  Objective  A: 

Establish  a  Western  Style  Military  Justice  System  That  Recognizes  the  Rights  of 

the  Individual 

Specific  Attributes: 

1 .  Established  legal  system  based  on  the  assumption  of  innocence,  the  law  of 

evidence,  and  the  burden  of  proof. 

Indicators  of  Performance: 

•  A  new  or  revised  military  justice  code  is  adopted  that  contains 
written  rules  akin  to  the  U.S.  Uniform  Code  of  Military  Justice 
(UCMJ)  which  define  offenses,  authority  and  jurisdiction  require- 
ments, and  outline  procedures  for  the  legal  system. 

•  The  system  allows  for  representation  of  the  accused. 

•  Conscientious  objection  is  decriminalized. 

•  The  legal  system  controls  the  release  of  personal  information. 

•  A  professional  corps  of  legal  officers  is  established  within  the 
military  (i.e.,:  A  JAG  corps). 


'-'  This  is  done  since  many  countries  possessing  the  various  attributes  that  characterize  sub-objective  ~2 
often  achieve  sub-objective  #1  simultaneously.  One  could  argue  that  greater  civilian  control  over  the 
military,  coupled  with  a  U.S.  style  military  justice  system,  is  an  effective  way  to  eliminate  the  potential  for 
war  crimes. 
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•  Military  persons  are  afforded  the  same  protections  that  civilians 
have  in  the  country's  civilian  judicial  system. 

•  Appeals  are  allowed. 

•  Court  officers  are  both  military  officers  and  officials  of  the  court. 

2.  The  prosecutor,  defense  council,  investigator,  and  judge  have  legally 

established  roles. 

Note:     This  attribute  can  be  further  divided  into  the  following  three 
attributes: 

a.  Effective  Advocacy. 

Indicators  of  Performance: 


• 


The  accused  is  allowed  to  consult  with  a  defense  lawyer 
prior  to  the  trial. 

•  Military  and  civilian  legal  sectors  work  together  to  ensure 
the  rights  of  the  accused  are  not  violated. 

•  Trial  advocacy  clinics  are  established  for  military  lawyers. 

b.  Fair,  Impartial,  and  Authoritative  Judgements. 

Indicators  of  Performance: 

•  Judgements  of  the  court  are  carried  out. 

•  Human  rights  executive  training  seminars  are  established 
for  military  judges. 

•  Members  of  the  military  are  openly  willing  to  have  their 
cases  heard  before  a  military  court. 

c.  Official  criminal  investigative  procedures  recognize  the  rights  of 
the  accused  and  accepted  standards  for  the  search  and  seizure  of 
evidence. 
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Indicators  of  Performance: 

•  The  investigative  process  is  revised  recognizing  the  rights  of  the 
individual. 

•  Interrogation     and     the     potential     for     self-incrimination     are 
constrained  by  law. 

•  The  search  and  seizure  of  evidence  is  strictly  regulated. 
The  sentencing  process  is  fair  and  impartial. 

Indicators  of  Performance: 


• 


• 


Detention  practices  are  revised  recognizing  the  right  of  due  process 
and  the  assumption  of  innocence. 

Alternatives  for  punishment  exist  aside  from  prison.  These  include 
administrative  punishment,  nonjudicial  punishment,  and  reduction 
in  pay  and  grade. 


4.  Humane  military  prison  system. 

Indicators  of  Performance: 

•  No  torture  is  allowed. 

•  Rules  exist  that  enforce  the  humane  treatment  of  prisoners. 

•  Outside  agencies,  such  the  International  Red  Cross,  are  allowed  to 
inspect  prison  facilities. 

5.  Corruption  and  the  abuse  of  power  are  prevented  and  ethical  behavior  is 
promoted. 

Indicators  of  Performance: 

•  No  undue  command  influence  is  exercised  by  the  military. 

•  Tampering  with  the  judicial  process  is  made  illegal. 
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•  Rules  of  engagement  (ROE)  are  developed  during  military 
operations  that  respect  the  sanctity  of  human  life  and  the  role  of 
non-combatants. 

Additional  Sub-Objective  #/: 
Civilian  Control  Over  the  Military 
Specific  Attributes: 
1 .  Elected  officials  control  the  military  judicial  system. 

Indicators  of  Performance: 

•  The  military  justice  system  originated  with  the  civilian  authority 
that  controls  the  military. 

•  Members  of  the  military  have  the  right  to  bring  unresolved 
problems  to  the  attention  of  their  elected  representatives. 

Note:    The  remaining  attributes  for  this  additional  sub-objective  are  the 
same  as  those  described  in  the  second  section  of  this  chapter. 

The  complete  hierarchy  of  sub-objectives  and  specific  attributes  for  the 
improved  military  justice  systems  objective  can  be  found  in  Figure  9. 

b.  Limitations 

There  are  limitations  to  the  use  of  the  researcher's  performance  indices. 
The  chief  limitation  is  the  selection  of  the  U.S.  military  justice  system  as  the  model  of 
choice.  While  few  argue  that  the  U.S.  system  of  justice  is  widely  regarded  for  its  respect 
for  human  rights,  the  assumption  that  all  countries  can  or  must  adhere  to  the  same  judicial 
procedures  may  be  problematic.  Customs  and  traditions  may  limit  the  strict  application 
of  the  U.S.  judicial  model  in  some  countries.  Application  of  all  the  performance 
indicators  identified  in  this  study  may  not  be  possible  in  some  countries. 
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Additionally,  the  proposed  performance  indicators  for  improved  military 
justice  systems  only  measure  structural  and  process  changes,  not  behavioral  ones. 
Measuring  changes  in  behavior  involves  highly  subjective  processes  and  is  not  possible 
with  the  researcher's  proposed  indicators.  A  change  to  the  military's  justice  system  is, 
however,  the  first  step  toward  changing  the  judicial  culture  of  the  military. 

3.  Defense  Resources  Management 

The  third  E-IMET  objective  is  the  improvement  of  defense  resources 
management.  The  need  for  better  management  of  military  resources  was  recognized  long 
before  Congress  expanded  the  IMET  program  to  include  respect  for  civilian  control  over 
the  military  and  human  rights.  The  Defense  Resources  Management  Institute  (DRMI), 
for  example,  was  established  at  the  Naval  Postgraduate  School  in  1965  to  help  the  United 
States  and  its  allies  with  resources  management  education  (DRMI  Website,  1998).  It  is 
important  to  note  that  programs  such  as  those  offered  by  DRMI  do  not  teach  job-specific 
skills.  Instead,  they  focus  on  building  management  skills  that  participants  can  apply 
throughout  their  careers.  In  order  to  propose  performance  indicators  for  this  objective, 
however,  one  must  clearly  define  the  specific  attributes  of  defense  resources  manage- 
ment. 

a.         Defining  Defense  Resources  Management 

While  defining  civilian  control  and  humane  military  justice  is  fairly 
straightforward,  specifying  the  attributes  of  improved  resource  management  is  much 
more  difficult.  To  date,  neither  the  State  Department  nor  DSCA  has  clearly  defined  the 
E-IMET  objective  of  "improved  defense  resources  management."  While  the  management 
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literature  does  attempt  to  define  the  domain  of  resources  management  (see  Hitch  and 
McKean  1967,  Sandler  and  Hartley  1995)  DSC  A  and  the  Department  of  State  have  not 
specified  the  final  result  desired  (also  known  as  "endstate")  for  their  training  and 
education  efforts  (Bonsper  Interview,  1998). 14  In  other  words,  while  the  endstates  of  the 
first  two  E-IMET  objectives  is  clear  (namely  civilian  dominance  over  the  military  and  the 
adoption  of  a  Western-style  military  justice  system)  the  final  result  desired  for  the 
improved  defense  resources  management  objective  is  ambiguous.  Without  a  clearly 
defined  endstate  one  cannot  identify  the  specific  attributes  of  defense  resources 
management. 

In  order  to  develop  the  objectives  hierarchy  and  performance  indices  for 
the  third  E-IMET  objective  the  researcher  proposes  the  following  endstate  for  improved 
defense  resources  management  training  and  education: 

The  adoption  of  a  decision-making  process  for  allocating  scarce 
defense  resources  that  is  transparent,  interactive,  and  objectives- 
oriented. 

The  researcher  defines  a  "transparent' '  decision-making  process  as  one  where  critical 
decision-making  steps  are  visibly  linked,  iterative,  and  follow  a  specific  schedule,  have  a 
defined  product,  are  completed  by  a  specified  stakeholder  or  group  of  stakeholders  (and 
the  completing  entity  is  known  to  all  the  remaining  stakeholders),  and  employ  a  well- 
defined  and  widely  understood  methodology.   The  opposite  of  a  transparent  process  is  a 


'4  The  final  result  desired  (or  "endstate")  describes  the  desired  result  of  the  action  (United  States  Marine 
Corps  Warfighting,  1989).  A  clear  explanation  and  understanding  of  the  desired  final  result  of  defense 
resources  management  training  and  education  is  absolutely  essential  to  the  development  of  specific 
attributes  and  subsequent  performance  indices. 
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"black  box"  where  defense  programs  are  the  outcome  of  some  unknown  military  planning 

process  (Bonsper,  1998).  A  decision-making  process  is  "interactive"  if  both  military  and 

civilian  stakeholders  play  an  active  role  during  each  step  in  the  process.    Finally,  it  is 

"objectives-oriented"  if  it  evaluates  existing  programs  and  develops  new  ones  based  on 

continuously  revised  strategic  objectives. 

Using  the  researcher's  proposed  endstate,  the  sub-objectives  and  specific 

attributes  for  the  E-IMET  objective  can  now  be  identified  and  arranged  into  an  objectives 

hierarchy. 

b.  Measuring  the  Performance  of  Defense  Resource  Management 

Programs 

The  E-IMET  objective  of  improved  defense  resource  management  can  be 
broken  down  into  the  following  three  sub-objectives: 

Sub-Objective  #1: 
Transparent  Resource  Management  Process 

Sub-Objective  #2: 

Interactive  Resource  Management  Process 

Sub-Objective  #3: 

Objectives-oriented  Resource  Management  Process 

The  following  specific  attributes,  along  with  their  respective  indicators  of 
performance,  have  been  identified  for  the  sub-objectives  of  improved  defense  resources 
management: 
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Sub-Objective  #1: 

Transparent  Resource  Management  Process 

Specific  Attributes: 

1.  Decision-making  steps  are  visibly  linked,  iterative,  and  follow  a  specific 
schedule. 

Indicators  of  Performance 

•  A  comprehensive  threat  assessment  is  performed  and  precedes 
defense  resource  allocation  decisions. 

•  A  national  military  strategy  is  developed  to  counter  the  anticipated 
threat  and  precedes  the  development  of  specific  programs. 

•  Defense  programs  are  developed  to  carry  out  the  national  military 
strategy. 

•  Defense  program  requirements  are  translated  into  annual  budget 
requests. 

•  Threat  assessments,  strategic  planning,  and  program  development 
occur  periodically  according  to  a  published  master  schedule. 

2.  Each  decision-making  step  results  in  a  defined  product. 

Indicators  of  Performance 


• 


• 


• 


Threat  assessments  result  in  official  strategic  planning  guidance 
issued  to  the  military'. 

A  national  military  strategy  is  published  that  defines  strategic 
objectives  and  priorities. 

Defense   programs  consist   of  detailed  manpower  (civilian   and 
military)  and  material  (equipment)  requirements. 

An  annual  budget  is  produced. 


ir, 


3.  Decision-making  steps  are  completed  by  a  specified  stakeholder  or  group 

of  stakeholders. 


Indicators  of  Performance 


• 


• 


• 


Civilian    and    military    intelligence    communities    cooperate    to 
produce  threat  assessments. 

Individual  service  chiefs  provide  input  to  the  national  military 
strategy. 

Defense  programs  are  developed  independently  by  each  service, 
but  are  reviewed  by  a  central  authority. 

Military  comptrollers  translate  program  requirements  into  annual 
budget  requests. 

The  people's  elected  representatives  approve  disbursements  from 
the  national  treasury  for  defense  programs. 

Civilian   authorities   approve   the   transfer   of  resources    among 
programs. 


4.  Each  step  in  the  decision-making  process  employs  a  well-defined  and 

widely  understood  methodology. 

Indicators  of  Performance 

•  Threat  assessments  update  past  threats,  reassess  current  threats,  and 

explore  future  threats. 


• 


• 


The  national  military  strategy  defines  the  military's  plan  for 
countering  current  and  future  threats  and  is  not  fiscally  constrained. 

Defense  program  choices  are  compared  to  each  other  on  the  basis 
of  cost  and  effectiveness  (capability)  and  ranked  in  order  of 
importance  (priority). 

The  planning  process  is  capable  of  determining  the  relative 
importance  of  defense  programs. 

Criteria  exist  to  evaluate,  rank,  and  prioritize  programs. 
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•  Civilian  and  military  leaders  use  cost  estimations  and  capability 
assessments  to  compare  individual  programs  before  selecting  a 
preferred  alternative. 

•  Budget  requests  are  compared  to  resource  availability. 

•  Priority  programs  are  funded  first  when  military  program  requests 
exceed  resource  availability  (and  funding  cuts  start  with  the  lowest 
priority  programs). 

Sub-Objective  #2: 

Interactive  Resource  Management  Process 

Specific  Attribute: 

1 .  Military  and  civilian  leaders  share  in  the  responsibilities  of  the  resource 

management  process. 

Indicators  of  Performance 

•  Military  officials  assess  the  threat,  develop  the  national  military 
strategy,  estimate  the  requirements  of  that  strategy,  and  develop 
programs  to  package  and  execute  the  strategy. 

•  The  people's  elected  representatives  approve  the  programs  along 
with  their  respective  cost  ceilings. 

•  Inter-agency  committees  composed  of  military  and  civilian 
officials  exist  to  coordinate  planning  and  budgeting  (i.e.:  a 
"corporate"  planning  board  overseas  the  entire  resource 
management  process). 

Sub-Objective  #3: 
Objectives-oriented  Resource  Management  Process 
Specific  Attributes: 
1 .  Individual  defense  programs  can  be  linked  to  specific  strategic  objectives. 
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Indicators  of  Performance 

The  military  develops  a  strategic  plan. 

Defense  programs  are  linked  to  objectives  within  the  military's 
strategic  plan. 

The  military  justifies  its  major  programs  to  the  people's  elected 
representatives  on  the  basis  of  strategic  objectives. 


2.  The  resource  management  process  balances  short-term  objectives  with 

long-term  objectives. 

Indicators  of  Performance 


• 


• 


• 


• 


• 


Civilian  officials  allocate  resources  to  maintain  ongoing  operations 
as  well  as  to  finance  investments  in  procurement,  new  weapons 
research,  and  military  construction. 

Long-term  investment  programs  (procurement,  construction, 
RDT&E)  are  fully  funded  while  ongoing  operations  are 
incrementally  funded. 1 5 

The  development  of  future  programs  and  the  formulation  of  the 
next  budget  occur  simultaneously  ("parallel  processes")  within  the 
resource  management  process. 

Civilian  and  military  officials  consider  long-term  trends  in  threat 
conditions  before  making  immediate  program  changes. 

The  level  of  military  program  funding  rises  and  falls  with  the  threat 
environment. 


Defense  programs  are  reevaluated  when  strategic  objectives  are  revised. 


• 


*5  "Fully  funded"  means  that  the  civilian  appropriators  have  provided  all  the  resources  necessary  to 
complete  the  approved  long-term  project  and  are  committed  to  seeing  the  project  through  to  its  completion. 
It  does  not  mean  that  the  military  receives  all  the  resources  at  once,  however.  "Incremental  funding"  is  the 
annual  appropriation  of  only  one  year's  worth  of  resources. 
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Indicator  of  Performance 

•     Periodic  cost-effectiveness  evaluations  of  existing  defense  programs  based  on 
revised  strategic  objectives. 

The  complete  hierarchy  of  sub-objectives  and  specific  attributes  for  the 
improved  defense  resources  management  objective  can  be  found  in  Figure  10. 

c.  Limitations 

Like  the  first  two  E-IMET  objectives,  there  are  limitations  to  the  use  of  the 
researcher's  performance  indices.  One  possible  limitation  is  the  use  of  a  generic 
Planning,  Programming,  and  Budgeting  System  (PPBS)  model  as  the  template  for  this 
analysis.  The  PPBS  template  provides  a  past  definition  of  defense  resource  management. 
Even  the  United  States,  considered  a  leader  in  defense  resource  management  initiatives, 
has  used  more  recent  processes  to  manage  its  defense  resources,  including  Management 
by  Objectives  (1973)  and  Zero-Base  Budgeting  (1977)  (GAO,  1997).  Although  the  PPB 
system  has  its  detractors  and  flaws  (see  Roberts  1993),  it  is  useful  to  demonstrate  how 
objectives  can  be  derived  even  if  one  were  to  substitute  another  process  for  PPBS  in  the 
future.  Therefore,  even  though  the  use  of  a  generic  PPBS  model  may  be  a  limitation,  the 
researcher  believes  it  is  an  excellent  starting  point  for  evaluating  defense  resources 
management. 

A  more  important  limitation  involves  the  actual  measurement  of 
progressions  or  improvements  away  from  the  defense  resources  management  baseline 
assessment.  In  order  to  observe  the  indicators  of  performance  security  assistance  officials 
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would  have  to  gain  access  to  the  highest  levels  of  all  ministries  where  defense  resource 
allocation  decisions  are  made.  In  the  absence  of  such  direct  high-level  access,  actual 
improvements  in  defense  resources  management  could  be  inferred  from  informal 
discussions  with  government  officials  at  all  levels,  public  statements  regarding  defense 
matters,  and  general  coverage  in  the  media  of  the  country. 

D.         THE  BALANCED  SCORECARD 

Assuming  a  country  receives  training  in  all  three  E-IMET  areas,  collecting  data  on 
every  indicator  will  likely  be  time  consuming,  expensive,  and  overwhelming  for  the 
security  assistance  manager.  In-country  security  assistance  officials  must  choose  a 
handful  of  indicators  to  apply  to  their  unique  situation.  In  order  to  present  the  various 
indicators  of  performance  for  each  E-IMET  objective  in  a  user-friendly  manner  the 
researcher  proposes  the  ''balanced  scGreGaTd"  format  developed  by  Kaplan  and  Norton. 

1.  The  Kaplan/Norton  Balanced  Scorecard 

The  Balanced  Scorecard  is  simply  %ia  set  of  measures  that  gives  top  managers  a 
fast  but  comprehensive  view  of  the  business"  (Kaplan  and  Norton,  1992,  p.  1).  Kaplan 
and  Norton  developed  the  Balanced  Scorecard  after  realizing  that  senior  private  sector 
executives  do  not  rely  on  only  one  set  of  performance  measures  to  the  exclusion  of 
another  (Kaplan  and  Norton,  1992). 

No  single  measure  can  provide  a  clear  performance  target  or  focus 
attention  on  the  critical  areas  of  the  business.  Managers  want  a  balanced 
presentation  of  both  financial  and  operational  measures  (Kaplan  and 
Norton,  1992,  p.  71). 
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Think  of  the  Balanced  Scorecard  as  the  dials  and  indicators  in  an  airplane 
cockpit.  For  the  complex  task  of  navigating  and  flying  an  airplane,  pilots 
need  detailed  information  about  many  aspects  of  the  flight.  [They  need] 
indicators  that  summarize  the  current  and  predicted  environment.  (Kaplan 
and  Norton,  1992,  p.  72). 

Private  sector  managers  realize  that  they  cannot  absorb  the  information  generated 
by  every  conceivable  measure  of  performance.  "The  Balanced  Scorecard  brings  together, 
in  a  single  management  report,  many  of  the  seemingly  disparate  elements  of  a  company's 
competitive  agenda"  (Kaplan  and  Norton,  1992,  p.  73).  The  Scorecard  forces  managers 
to  choose  among  various  performance  measures  with  different  business  perspectives 
(objectives),  focusing  on  the  handful  of  measures  that  are  most  critical  (Kaplan  and 
Norton,  1992).  By  selecting  three  to  four  key  measures  within  each  business  objective 
managers  can  link  a  wide  range  of  performance  measurement  data  allowing  them  to  make 
better  decisions  in  a  time  critical  environment.  The  format  of  the  Kaplan/Norton 
Balanced  Scorecard,  as  well  as  their  private  sector  example  of  an  electronics  company, 
can  be  found  in  Figure  1 1 . 

2.  The  E-IMET  Balanced  Scorecard 

While  Kaplan  and  Norton  use  the  Balanced  Scorecard  to  present  financial  and 
operational  data  for  private  sector  firms,  the  same  method  of  presentation  is  useful  for 
managers  of  public  sector  programs  with  multiple  objectives.  The  Kaplan/Norton 
Balanced  Scorecard  format  can  be  modified  so  that  chosen  performance  indicators  are 
listed  alongside  their  respective  sub-objective  each  of  the  E-IMET  objectives.  These 
modified  scorecards  can  serve  as  comprehensive  security  assistance  management  tools 
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Format: 


Perspective 

GOALS 

Measures 

Electronics  Company  Example: 


FINANCIAL  Perspective 

GOALS 

Measures 

Survive 

Cash  Flow 

Succeed 

Quarterly- 

Sales  Growth 

Operating 

Income 

Prosper 

Increased  ROE 

CUSTOMER  Perspective 

GOALS 

Measures 

Responsive 
Supply 

New  Products 

Preferred 
Supplier 

On-Time  Delivery 

Percent  of  Sales 
From  New 
Products 

Share  of  Key 

Accounts' 

Purchases 

\ 


/ 


INTERNAL  BUSINESS  Perspective 

GOALS 

Measures 

Manufacturing 

Cycle  Time 

Excellence 

Unit  Cost 

Yield 

Design 

Silicon  Efficiency 

Productivity 

Engineering  Efficiency 

Figure  11.  Kaplan/Norton  Balanced  Scorecard  Format  and  Example 
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for  tracking  the  achievement  of  each  E-IMET  objective  within  a  given  country.  The 
Kaplan/Norton  private  sector  Balanced  Scorecard  thus  becomes  the  "(Country's  Name) 
Balanced  Scorecard"  for  the  E-IMET  program. 

The  researcher  recommends  that  the  selection  of  performance  indicators  for  a 
specific  country  be  done  by  State  or  Defense  Department  professionals  with  significant 
in-country  experience  (i.e.:  A  Security  Assistance  Officer  (SAO)  or  other  embassy 
official).  Deciding  what  indicators  of  performance  are  most  meaningful  for  their  country 
is  a  subjective  process.  For  example,  to  aid  in  the  selection  of  the  handful  of  civilian 
control  indicators  used  within  a  specific  country  Paul  Stockton  of  the  Naval  Postgraduate 
School's  Center  for  Civil-Military  Relations  recommends  grouping  the  various  attributes 
and  performance  indicators  into  two  major  categories;  those  that  are  essential  to  civilian 
control  and  those  that  are  indicators  of  the  quality  or  depth  of  civilian  control.16  Since  no 
in-country  security  assistance  officials  participated  in  this  study,  an  accurate  grouping 
according  to  Dr.  Stockton's  recommendation  is  outside  the  scope  of  this  study.  Such  a 
categorization  would,  however,  help  regional  security  assistance  officers  decide  what 
performance  indicators  to  include  in  their  country's  balanced  scorecard. 

A  complete  "(Country's  Name)  Balanced  Scorecard"  example  can  be  found  in 
Figure  12. 


1"  Dr.  Stockton  identifies  a  third  category  as  well;  contentious  indicators  of  civilian  control.  This  category 
includes  controversial  indicators  that  are  often  debated  among  civil-military  relations  scholars.  The 
researcher  discussed  this  category  in  the  indicators  of  civilian  control  over  the  military  limitations  section. 
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CIVILIAN  CONTROL  OVER  THE  MILITARY 

SUB -OBJECTIVE 

Indicators    of 

Performance 

Military  Used 

Implementation  of 

to  Counter 

Military  Doctrines 

External 

Stressing  External 

Threats 

Roles  and  Missions 

Redeployment  of 

Military  Away  From 

Internal  Uses 

Revised  curriculum 

at  service 

academies 

IMPROVED  MILITARY  JUSTICE  SYSTEMS 

ADDITIONAL 

Indicators    of 

OBJECTIVE 

Performance 

Establish  a 

Appeals  are  Allowed 

Western 

Style 

Military 

The  Accused  is 

Justice 

Allowed  to  Consult 

System 

With  a  Defense 
Lawyer 

The  Investigative 
Process  is  Revised 
Recognizing  the 
Rights  of  the 
Individual 

\     / 


IMPROVED  DEFENSE  RESOURCE 

MANAGEMENT 

SUB -OBJECTIVE 

Indicators    of 

Performance 

Decision-making 

A  comprehensive 

Steps  are 

threat  assessment 

Visibly  linked, 

is  performed 

Iterative,  and 

Follow  a  specific 

A  national 

Schedule 

military  strategy 

Is  developed 

Defense  programs 

Are  developed 

Based  on  the 

National  military 

Strategy 

Figure  12. 


'(Country's  Name)  Balanced  Scorecard"  Example 
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CHAPTER  SUMMARY 


Objectively  measuring  the  results  of  the  E-IMET  effort  involves  tracking  the 
changes  within  a  country  toward  greater  civilian  control  over  the  military,  more  humane 
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military  justice  systems,  and  improved  defense  resources  management.  In  order  for 
officials  to  determine  whether  the  E-IMET  intervention  is  responsible  for  significant 
organizational  and  national  changes  a  baseline  assessment  must  first  be  compiled.  Due  to 
the  multi-faceted  nature  of  resource  management,  the  State  Department  and  DSCA  must 
also  clearly  define  a  desired  endstate  for  "improved  defense  resources  management." 
Various  indicators  of  performance  can  then  be  combined  with  survey  data  so  security 
assistance  officers  can  compare  post-training  changes  with  the  baseline  assessment  to  get 
a  clearer  picture  of  the  effectiveness  of  E-IMET  training.  While  limitations  do  exist,  the 
performance  indicators  proposed  in  this  chapter  offer  an  alternative  to  traditional 
anecdotal  evidence. 
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VII.      CONCLUSION 

A.         INTRODUCTION 

In   1991,  the  U.S.  Congress  expanded  the  scope  of  the  International  Military 
Education  and  Training  (IMET)  program  to  include  a  focus  on  (Cope,  1995): 


• 


Greater  respect  for  and  grasp  of  democracy  and  civilian  rule  of  law, 
including  the  principle  of  civilian  control  over  the  military; 

•  Military  justice  systems  in  a  democracy,  including  a  better  understanding 
of  internationally  recognized  human  rights; 

•  Responsible  defense  resource  management. 

In  doing  so,  they  aligned  the  IMET  program  with  the  new  Post  Cold  War  security 
environment.  The  Departments  of  State  and  Defense,  however,  did  not  agree  on  a  system 
to  evaluate  the  effectiveness  of  the  expanded  IMET  (E-IMET)  program.  In  1993.  the 
U.S.  Congress  enacted  the  Government  Performance  and  Results  Act  requiring  all 
government  agencies  to  establish  top-level  goals  and  objectives,  define  how  they  intend 
to  achieve  those  goals,  and  demonstrate  how  they  will  measure  agency  and  program 
performance  in  achieving  those  goals  (GPRA,  1993). 

The  purpose  for  this  study  is  to  address  how  the  Defense  Security  Cooperation 
Agency  (DSCA)  and  the  Department  of  Defense  (DoD)  can  tackle  the  challenge  of 
measuring  the  effectiveness  and  performance  of  the  E-IMET  program.  This  chapter  will 
summarize  the  study's  findings  based  on  the  original  research  questions. 
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B.  RESTATEMENT  OF  RESEARCH  QUESTIONS 

Primary  Research  Question: 

What  could  a  performance  evaluation  system  developed  for  the  E-IMET  program 
look  like? 

Subsidiary  Research  Questions: 

1 .  What  are  the  challenges  to  performance  evaluation  of  public  programs? 

2.  What  factors  unique  to  the  political  environment  necessarily  limit  the 
application  of  rational -analytic  assessment  models  when  measuring  costs 
and  benefits  of  current  security  assistance  programs? 

3.  Can  a  system  be  developed  that  balances  the  reality  of  the  political 
bureaucratic  process  with  the  rationality  of  economic  cost-benefit 
analysis? 

4.  What  are  the  implications  of  implementing  a  performance  evaluation 
system  on  current  security  assistance  programs? 

C.  SUMMARY  OF  FINDINGS 

1.  Challenges  to  Public  Sector  Performance  Evaluation 

The  obstacles  to  public  sector  performance  evaluation  are  primarily  institutional, 
technical,  and  financial.  Each  of  these  obstacles  presents  a  significant  challenge  to  E- 
IMET  performance  evaluation. 

a.         Institutional  Obstacles 

Institutional  obstacles  include  the  influence  of  political  pressure  on 
performance  measurement,  the  effect  of  multiple  self-interested  constituencies  on 
objectives  consensus,  and  the  lack  of  experience  of  government  managers  in  dealing  with 
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performance  data.  Each  of  these  obstacles  limits  the  ability  of  E-IMET  officials  to 
measure  the  effectiveness  of  the  E-IMET  program. 

The  E-IMET  program  has  four  diverse  groups  of  stakeholders,  all  sharing 
in  the  management  of  the  E-IMET  program.  The  presence  of  so  many  powerful 
stakeholder  groups  makes  the  setting  of  a  single  set  of  security  assistance  objectives 
difficult.  The  Executive  Office  of  the  President  sets  national  security  assistance  direction 
while  the  Congress,  driven  by  their  commitment  to  the  Government  Performance  and 
Results  Act,  appropriates  the  funds  necessary  for  the  programs.  Each  has  a  different 
political  stake  in  the  performance  of  the  E-IMET  program.  The  Departments  of  State  and 
Defense  share  responsibility  for  the  management  of  the  E-IMET  program,  but  have 
potentially  different  financial  stakes  in  the  measurement  of  the  program's  effectiveness. 
Getting  these  groups  to  agree  on  a  single  set  of  objectives  for  security  assistance  and  the 
E-IMET  program  is  a  challenge. 

Compounding  this  problem  is  the  fact  that  State  and  Defense  Department 
official  documents  do  not  separate  objectives  from  strategies.  Fortunately,  once  the 
researcher  separated  objectives  from  strategies  a  common  set  of  national  security 
assistance  objectives  emerged.  This  consensus  forms  the  objective  baseline  for  future 
performance  evaluation. 

b.  Technical  Obstacles 

Technical  obstacles  include  differentiating  between  output  and  outcome 
measures,  the  need  for  several  measures  of  performance  for  any  one  program,  the  lack  of 
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standards  for  judging  governmental  performance,  and  the  lack  of  timely  performance 
feedback. 

Due  to  the  multi-faceted  nature  of  the  E-IMET  program  no  single  measure 
of  performance  is  adequate.  E-IMET  officials  must  develop  and  implement  multiple 
measures  of  performance  simultaneously  in  order  to  provide  Congress  with  the  data 
required  by  the  Results  Act.  Even  with  such  a  system,  no  standards  exist  that  indicate 
whether  the  E-IMET  intervention  is  "effective  enough"  to  merit  increased  funding. 
Periodic  performance  reporting  must  be  compared  with  an  objective  baseline  assessment 
of  a  country's  political,  judicial,  and  managerial  characteristics  (collected  before  the  first 
E-IMET  intervention)  in  order  to  attribute  specific  results  to  the  E-IMET  intervention. 

Additionally,  security  assistance  programs  like  E-IMET  are  often  long- 
term  intervention  efforts.  Such  efforts  often  do  not  produce  immediate  results  and  are 
therefore  not  conducive  to  the  immediacy  of  the  GPRA  reporting  requirement. 

c.  Financial  Obstacles 

Implementation  of  performance  measurement  systems  can  be  very  costly, 
stretching  the  availability  of  financial  and  human  resources.  Systematically  tracking  the 
progression  toward  civilian  control  over  the  military,  Western-style  military  justice 
systems,  and  improved  defense  resource  management  within  a  country  requires  the 
commitment  of  State  and  Defense  Department  assets.  Funding  for  the  IMET  and  E- 
IMET  programs  is  already  tiny  relative  to  other  discretionary  programs.  Implementing  a 
comprehensive  E-IMET  performance  evaluation  system  would  likely  absorb  the  very 
funding  increases  that  the  system  is  designed  to  capture.   A  comprehensive  performance 
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measurement  system  implemented  under  current  budgetary  constraints  would  be  difficult. 
DoD  and  DSCA  must  therefore  adopt  a  performance  evaluation  system  that  is  easy  to  use 
and  can  be  implemented  without  additional  funding. 

2.  Factors  Limiting  Rational-Analytic  Assessment 

The  chief  factor  limiting  the  objective  assessment  of  E-IMET  effectiveness  is  the 
inability  to  control  external  variables.  It  is  difficult  to  separate  the  effects  of  the  external 
environment  from  the  E-IMET  intervention.  Security  assistance  officials  will  likely 
never  be  able  to  establish  a  concrete  cause  and  effect  relationship  between  the  E-IMET 
intervention  and  a  given  result  due  to  the  presence  of  uncontrollable  political  and 
economic  forces.  They  can,  however,  do  a  better  job  demonstrating  to  Congress  a 
country's  post-intervention  progression  toward  the  objectives  of  the  E-IMET  program. 

Technically  measuring  the  effects  of  the  E-IMET  program  is  also  difficult. 
Quantifying  the  benefits  that  result  from  greater  civilian  control  over  the  military, 
improved  military  justice  systems,  and  more  responsible  defense  resource  management  is 
very  subjective.  Further  complicating  this  process  is  the  lack  of  a  clear  definition  of 
improved  defense  resources  management.  Neither  DSCA  nor  the  Department  of  State 
has  clearly  defined  a  desired  endstate  for  the  improved  defense  resource  management 
objective.  While  the  attributes  of  civilian  control  over  the  military  and  Western-style 
military  justice  systems  are  much  easier  to  describe,  the  attributes  of  improved  defense 
resource  management  are  more  ambiguous. 

Another  important  limitation  to  the  performance  measurement  process  is  the 
disjointedness  of  the  government's  security  assistance  organizations.    In  addition  to  the 
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obstacles  created  by  the  presence  of  multiple  constituencies,  the  current  bureaucratic 
structure  of  the  security  assistance  community  is  a  major  impediment  to  the  measuring 
process.  The  researcher  agrees  with  a  1995  National  Defense  University  study  that 
observed: 

The  apportionment  of  responsibilities — system  oversight  (State),  program 
management  (Defense),  and  program  requirements  (Unified  Commands, 
program  teams,  and  foreign  governments)... is  inherently  dysfunctional, 
creating  the  impression  that  IMET  is  a  program  without  long-range 
benefit,  one  well-suited  only  for  near-term  displays  of  U.S.  attention  and 
associated  "carrot  and  stick"  diplomacy.  This  leads  many  in  government 
to  conclude  that  IMET  is  an  expendable  program  (Cope,  1995,  pp.  52-53). 

The  report  added  that: 

Although  international  military  education  and  training  is  a  diplomatic- 
defense,  or  political-military,  instrument  of  national  security  policy, 
INSS's  research  found  substantial  agreement  that  the  DoD  has  the  greatest 
interest  in  and  commitment  to  the  IMET  program.  For  the  last  19  years, 
the  State  Department  has  dominated  the  process,  while  remaining 
ambivalent  toward  it,  whereas  Defense,  without  active  sustained  interest 
by  staff  principles,  generally  has  deferred  to  State  (Cope,  1995,  p.  53). 

The  results  of  this  study  show  that  little  has  changed  within  the  State  and  Defense 
Departments  since  1995.  Sharing  responsibility  for  the  IMET  program  and  its  expanded 
initiative  between  two  major  bureaucratic  government  agencies  guarantees  that  perform- 
ance measurement  will  fail  to  achieve  its  potential.  Until  the  Departments  of  State  and 
Defense  streamline  the  program's  chain  of  responsibility  and  authority  E-IMET 
performance  evaluation  will  continue  to  be  limited. 


110 


3.  The  E-IMET  Performance  Evaluation  System 

A  spectrum  of  approaches  to  the  evaluation  of  public  programs  exists  within  the 
performance  evaluation  literature.  Anchoring  one  end  of  the  spectrum  is  the  "technically 
rational  paradigm"  that  treats  performance  measurement  as  an  objective,  scientific  aid  to 
decision-making.  Anchoring  the  opposite  end  of  the  spectrum  is  the  "politically  rational 
paradigm"  that  emphasizes  the  effect  of  political  power  on  the  measurement  of 
performance.  Given  the  difficulties  of  technically  measuring  the  effects  of  the  E-IMET 
program,  the  politically  rational  paradigm  was  used  to  develop  the  following  four-step  E- 
IMET  performance  measurement  process: 

1 .  Identify  the  Key  E-IMET  Stakeholders; 

2.  Identify  the  Objectives  of  U.S.   Security  Assistance  and  the  E-IMET 
Program; 

3.  Develop  Performance  Indices  for  the  E-IMET  Objectives; 

4.  Apply  technically  rational  performance  measurement  techniques. 

Only  steps  #1-3  were  completed  during  this  study.  Due  the  researcher's  lack  of 
access  to  country-specific  E-IMET  training  data  and  security  assistance  officials,  step  #4 
was  beyond  the  scope  of  this  study.  In  order  to  apply  the  E-IMET  performance  indices 
proposed  in  Chapter  VI,  DCSA  must  first  conduct  an  objective  baseline  assessment  in 
each  country  that  receives  E-IMET  training.  DSCA  must  then  allow  the  in-country 
Security  Assistance  Officers  (S  AOs)  to  select  the  indicators  of  performance  that  are  most 
applicable  to  their  country's  current  situation.   Systematic  reporting  of  progressions  away 


1  1 


from  the  baseline  assessment  must  follow  in  order  to  establish  a  stronger  cause  and  effect 
relationship  and  meet  the  requirements  of  the  Results  Act. 

4.  Implications  of  the  E-IMET  Performance  Evaluation  System 

Successfully  measuring  the  performance  of  the  E-IMET  program  has  both  internal 
and  external  implications  for  organizations  like  DSCA. 

a.  Internal  Implications 

Requiring  SAOs  to  systematically  report  on  the  progress  toward  greater 
civilian  control  over  the  military,  improved  military  justice  systems,  and  improved 
defense  resources  management  is  critical  to  the  performance  measurement  effort. 
Information  is  only  as  good  as  its  source,  however.  SAOs  must  be  trained  to  recognize 
the  indicators  of  performance  and  disciplined  enough  to  consistently  report  them. 
Additionally,  DSCA  must  be  organized  to  receive  the  data  and  possess  the  processing 
capabilities  necessary  for  the  collection  and  dissemination  of  the  information. 

To  date,  security  assistance  officials  have  only  been  required  to  present 
anecdotal  evidence  to  guarantee  a  future  for  their  programs.  Informal  student  opinions, 
the  testimony  of  a  key  Unified  Commander,  and  the  remarks  of  an  Ambassador  or  foreign 
official  are  examples  of  the  anecdotal  evidence  used  by  security  assistance  officials  to 
justify  their  programs.  But  the  Results  Act  requires  organizations  like  DSCA  to 
demonstrate  the  results  achieved.  The  application  of  a  rigorous  performance 
measurement  system  may  be  outside  the  experience  of  most  security  assistance  managers. 
Understanding  the  limitations  and  challenges  of  a  public  sector  performance  evaluation 
system  is  crucial  to  making  one  work.    Performance  measurement  systems  like  the  one 
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designed  in  this  study  must  be  fully  integrated  with  the  strategic  management  of  an 
organization.  Inexperienced  security  assistance  officials  might  see  the  results  of  their 
performance  evaluation  efforts  simply  as  a  report  card  and  not  as  the  strategic 
management  tools  that  they  are.  Successfully  evaluating  the  performance  of  an  E-IMET 
intervention  will  determine  if  it  is  having  the  intended  effect  but  it  will  not  tell  DSCA 
officials  how  to  change  it  to  make  it  more  effective. 

b.  External  Implications 

The  external  implications  of  measuring  the  performance  of  security 
assistance  programs  are  mixed.  If  the  results  are  not  favorable  should  one  conclude  that 
efforts  to  export  democratic  ideals  are  failing  and  should  be  stopped  in  order  to  save  only 
3%  of  the  federal  budget?  Exporting  democracy  through  security  assistance  training  has 
traditionally  been  an  effort  based  on  faith— "train  them  and  they  will  change." 
Demonstrating  results  is  foreign  to  the  nature  of  security  assistance  program  managers 
who  do  not  discount  non-quantifiable  factors.  Meeting  the  requirements  of  the  Results 
Act  could  help  solidify  future  congressional  support  but  the  emphasis  on  visible  results 
might  overshadow  invisible  ones.  An  imperfect  E-IMET  performance  evaluation  system 
must  be  used  to  reinforce  the  testimony  of  international  affairs  experts,  not  as  a 
replacement  for  informed  opinion. 

D.         RECOMMENDATIONS 

1.  Recommendations  for  Further  Action 

It  is  recommended  that  DSCA  consider  implementing  the  use  of  the  performance 
indicators    proposed    in   this    study    to    measure    the   effectiveness   of  their    E-IMET 
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intervention  efforts.  To  do  so,  DSCA  and  in-country  security  assistance  officials  must 
build  a  country-specific  Balanced  Scorecard  of  performance  indicators  for  each  of  the  E- 
IMET  objectives.  A  standard  baseline  assessment  format  must  also  be  developed  to 
provide  the  comparative  foundation  from  which  to  measure  the  effectiveness  of  the  E- 
IMET  intervention.  The  training  of  the  S  AOs  to  recognize  the  indicators  of  performance 
and  properly  complete  the  baseline  assessment  must  be  a  prerequisite  of  the  system's 
implementation.  Information  resulting  from  the  systematic  reporting  of  progressions 
away  from  the  baseline  must  then  be  gathered  over  time  and  presented  as  evidence  for 
future  funding  decisions. 

The  researcher  also  recommends  that  DSCA  and  the  Department  of  State  clearly 
define  an  endstate  for  "improved  defense  resources  management."  A  final  result  desired, 
as  well  as  the  desired  attributes,  must  be  communicated  throughout  DSCA.  DSCA 
should  solicit  the  help  of  Naval  Postgraduate  School  academic  professionals  to  build 
upon  the  researcher's  definition  and  objectives  hierarchy  for  improved  defense  resources 
management. 

2.  Recommendations  for  Further  Study 

This  study  was  completed  without  the  input  of  regional  experts.  Building  a 
Balanced  Scorecard  of  E-IMET  performance  evaluation  indicators  requires  a  broad 
understanding  of  a  country's  civil-military  traditions,  existing  military  justice  system,  and 
defense  resource  management  practices.  The  indicators  of  performance  proposed  in  this 
study  must  be  refined  to  fit  the  unique  situations  of  specific  countries.     A  baseline 
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assessment  format  must  also  be  developed  so  that  future  progressions  toward  the 
objectives  of  the  E-IMET  program  can  be  measured. 

This  study  also  did  not  consider  alternatives  for  the  delivery  of  the  security 
assistance  service.  The  researcher  developed  performance  indices  for  the  program  as  it 
currently  exists.  Now  that  performance  indices  have  been  proposed  a  cost-effectiveness 
comparison  of  alternatives  is  possible.  Other  alternatives  for  delivering  security 
assistance  training  should  be  explored  to  determine  the  most  efficient  and  effective 
method. 

While  not  specifically  addressed  during  this  study,  there  are  implications  that  the 
current  apportionment  of  security  assistance  management  responsibilities  within  the 
national  government  is  inherently  dysfunctional  and  basic  IMET  relationships  need 
adjustment.  DoD  and  DoS  may  need  to  streamline  IMET  management  and  budget 
planning,  create  a  dedicated  interdepartmental  review  council  to  oversee  the  entire 
program,  and  review  the  overarching  policies  that  guide  foreign  military  education. 
Further  study  is  needed  to  determine  whether  current  organizational  relationships  are 
appropriate  and  to  determine  the  optimal  apportionment  of  IMET  responsibilities  and 
authority. 
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APPENDIX.  DEFINITIONS 

The  following  terms  have  been  defined  as  listed  below  for  the  purposes  of  this  study: 

Baseline  Data:  Initial  collection  of  data  to  establish  a  basis  for  comparison. 

Benchmark:  A  standard  or  point  of  reference  used  in  measuring  and/or  judging  quality 
or  value. 

Effectiveness:  The  production  of  a  decided,  decisive,  or  desired  effect. 

Efficiency:  The  ratio  of  outputs  to  inputs  in  a  process. 

Emerging  Democracy:  A  formerly  non-democratic  national  government  that  seeks  to 
implement  broad-based  democratic  reforms.  Examples  include  Poland,  Hungary,  and 
various  former  Soviet  Republics  such  as  Estonia,  Latvia,  and  Lithuania. 

External  Stakeholder:  An  individual  or  group  outside  the  boundaries  of  the  producing 
organization  or  program  that  receives  or  uses  the  output  of  the  process. 

Internal  Stakeholder:  An  individual  or  group  inside  the  boundaries  of  the  producing 
organization  or  program  that  receives  or  uses  the  output  from  a  previous  stage  or  process 
to  contribute  to  production  of  the  final  product  or  service. 

Measure:  One  of  several  measurable  values  that  contribute  to  the  understanding  and 
quantification  of  a  performance  indicator. 

Measurement  System:  A  system  of  practices,  procedures,  criteria,  and  standards  that 
govern  the  collection  of  data  (input),  the  analysis  of  the  data  (throughput),  and  the 
compilation  of  the  results  into  quantitative  or  qualitative  forms  (output). 

Mobile  Education  Teams  (METs):  A  group  of  American-led  professionals  tasked  to 
travel  to  a  host  country  to  conduct  E-IMET  courses.  It  should  be  noted  that  "METs" 
differ  from  the  more  commonly  known  military  *"MTTs'"  (Mobile  Training  Teams). 
METs  are  primarily  civilian  educators  that  teach  military-to-civilian  training  courses, 
where  MTTs  are  primarily  military  personnel  who  deliver  military-to-military  training. 

Objectives  Hierarchy:  A  useful  approach  to  systematically  link  key  stakeholders  and 
their  respective  objectives  to  performance  indicators.  A  hierarchy  works  down  from  a 
higher  level  objective  to  relevant  sub-objectives,  to  specific  attributes  that  characterize 
those  sub-objectives,  and  finally  to  measurable  characteristics  that  allow  for  the 
quantification  of  the  desired  attributes. 
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Outcome  Measurement:  An  assessment  of  the  results  of  a  program  activity  as 
compared  to  its  intended  purpose.  Outcome  refers  to  the  impact  on  society  of  a  particular 
public  sector  activity  and  is  closely  related  to  the  concept  of  effectiveness. 

Output  Measurement:  The  tabulation,  calculation,  or  recording  of  an  activity  or  effort. 
Output  measurement  traditionally  refers  to  the  quantity  of  the  good  or  service  provided, 
and  is  closely  related  to  the  concept  of  efficiency . 

Paradigm:  A  systematic  statement  of  the  basic  assumptions,  concepts,  and  propositions 
employed  by  a  school  of  analysis. 

Performance  Goal:  A  target  level  of  an  activity  expressed  as  a  tangible  measurable 
objective,  against  which  actual  achievement  can  be  compared. 

Performance  Indicator:  A  particular  value  or  characteristic  used  to  measure  output  or 
outcome. 

Performance  Management:  The  use  of  performance  measurement  information  to  help 
set  agreed-upon  performance  goals,  allocate  and  prioritize  resources,  inform  managers  to 
either  confirm  or  change  current  policy  or  program  directions  to  meet  those  goals,  and 
report  on  the  success  in  meeting  those  goals. 

Performance  Measure:  A  quantitative  or  qualitative  characterization  of  performance. 

Performance  Measurement:  A  process  of  assessing  progress  toward  achieving 
predetermined  goals,  including  information  on  the  efficiency  with  which  resources  are 
transformed  into  goods  and  services  (outputs),  the  quality  of  those  outputs  (how  well  they 
are  delivered  to  clients  and  the  extent  to  which  clients  are  satisfied),  and  outcomes  (the 
results  of  a  program  activity  compared  to  its  intended  purpose),  and  the  effectiveness  of 
government  operations  in  terms  of  their  specific  contributions  to  program  objectives. 

Performance-based  Budgeting:  Budgeting  initiatives  designed  to  measure  performance 
and  link  resources  with  results.  The  latest  iteration  in  a  series  of  Federal  Government 
performance  budgeting  reforms  is  the  Government  Performance  and  Results  Act  (GPRA) 
of  1993. 

Political  Rational  Paradigm:  A  paradigm  that  emphasizes  the  effect  of  political  power 
on  the  measurement  of  performance. 

Process-Oriented  Measurement  System:  A  measurement  system  whose  purpose  is  to 
assess  the  attainment  of  the  desired  results,  or  outcomes,  based  on  defined  agency 
objectives. 

Program  Performance  Measurement:  The  regular  collection  and  reporting  of  a  range 
of  data  including  inputs,  workload  levels,  outputs,  and  outcomes. 
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Results-Oriented  Measurement  System:  A  measurement  system  whose  purpose  is  to 
assess  the  efforts  to  reengineer  agency  processes  in  order  to  achieve  greater  efficiency. 

Security  Assistance:  The  transfer  of  military  and  economic  assistance  to  foreign 
governments  through  sale,  grant,  lease,  or  loan. 

Stake:  A  claim  on  the  organization  or  program  that  each  stakeholder  has.  A  stake  may 
be  tangible  or  intangible,  explicit  or  implicit.  A  stake  may  also  be  political,  economic, 
social,  or  psychological  in  nature. 

Stakeholder:  Any  person,  group,  or  organization  that  can  place  a  claim  on,  or  influence, 
the  organization's  or  program's  resources,  outputs,  or  outcome;  is  affected  by  those 
outputs;  or  has  an  interest  or  expectation  of  the  organization  or  program. 

Strategic  Goal:  A  long-range  change  target  that  guides  an  organization's  efforts  in 
moving  toward  a  desired  future  state  (also  referred  to  as  an  "end"). 

Strategic  Objective:  A  broad  time-phased  measurable  accomplishment  required  to 
realize  the  successful  completion  of  a  strategic  goal. 

Technical  Rational  Paradigm:  A  paradigm  that  treats  performance  measurement  as  an 
objective,  scientific  aid  to  decision-making.  Central  to  the  application  of  this  paradigm  is 
the  existence  of  objectives  consensus  and  measurable  costs  and  benefits.  Examples 
include  economic  cost-benefit  analysis,  performance  ratios,  and  linear  programming 
models. 
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